[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 459277578 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003053] x2apic enabled [ 0.003966] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007008] pid_max: default: 32768 minimum: 301 [ 0.008106] LSM: Security Framework initializing [ 0.009038] Yama: becoming mindful. [ 0.009909] SELinux: Initializing. [ 0.010000] *** VALIDATE selinux *** [ 0.016203] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021142] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024060] *** VALIDATE tmpfs *** [ 0.025214] *** VALIDATE proc *** [ 0.026149] *** VALIDATE cgroup *** [ 0.027004] *** VALIDATE cgroup2 *** [ 0.028005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029117] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031000] Spectre V2 : User space: Vulnerable [ 0.031005] Speculative Store Bypass: Vulnerable [ 0.033420] debug: unmapping init [mem 0xffffffff8ee59000-0xffffffff8ee60fff] [ 0.035159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036590] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037009] ... version: 2 [ 0.037803] ... bit width: 48 [ 0.038008] ... generic registers: 4 [ 0.038693] ... value mask: 0000ffffffffffff [ 0.039007] ... max period: 00007fffffffffff [ 0.040006] ... fixed-purpose events: 3 [ 0.040739] ... event mask: 000000070000000f [ 0.041305] rcu: Hierarchical SRCU implementation. [ 0.043589] smp: Bringing up secondary CPUs ... [ 0.044655] x86: Booting SMP configuration: [ 0.045022] .... node #0, CPUs: #1 #2 #3 [ 0.050149] smp: Brought up 1 node, 4 CPUs [ 0.051885] smpboot: Max logical packages: 1 [ 0.052009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.078000] node 0 deferred pages initialised in 26ms [ 0.081303] devtmpfs: initialized [ 0.083244] x86/mm: Memory block size: 128MB [ 0.085788] gcov: version magic: 0x41383552 [ 0.089253] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.093076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.095276] pinctrl core: initialized pinctrl subsystem [ 0.096079] [ 0.096421] ************************************************************* [ 0.098006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.101009] ** ** [ 0.103008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.105008] ** ** [ 0.108009] ** This means that this kernel is built to expose internal ** [ 0.110009] ** IOMMU data structures, which may compromise security on ** [ 0.113011] ** your system. ** [ 0.115010] ** ** [ 0.117010] ** If you see this message and you are not debugging the ** [ 0.119007] ** kernel, report this immediately to your vendor! ** [ 0.122010] ** ** [ 0.124009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127009] ************************************************************* [ 0.129845] NET: Registered protocol family 16 [ 0.131446] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.133038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.135032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.137652] cpuidle: using governor menu [ 0.140213] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.141463] PCI: Using configuration type 1 for base access [ 0.143083] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.154056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.155163] cryptd: max_cpu_qlen set to 1000 [ 0.156293] ACPI: Added _OSI(Module Device) [ 0.158055] ACPI: Added _OSI(Processor Device) [ 0.159008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.160023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.163309] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.167464] ACPI: Interpreter enabled [ 0.168041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.170055] ACPI: Using IOAPIC for interrupt routing [ 0.171057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.173296] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.184084] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.186027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.188016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.190066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.193541] acpiphp: Slot [2] registered [ 0.195093] acpiphp: Slot [5] registered [ 0.196138] acpiphp: Slot [6] registered [ 0.197081] acpiphp: Slot [3] registered [ 0.199068] acpiphp: Slot [4] registered [ 0.199950] acpiphp: Slot [7] registered [ 0.201044] acpiphp: Slot [8] registered [ 0.202017] acpiphp: Slot [9] registered [ 0.202987] acpiphp: Slot [10] registered [ 0.203049] acpiphp: Slot [11] registered [ 0.204054] acpiphp: Slot [12] registered [ 0.205010] acpiphp: Slot [13] registered [ 0.207060] acpiphp: Slot [14] registered [ 0.208066] acpiphp: Slot [15] registered [ 0.209083] acpiphp: Slot [16] registered [ 0.210060] acpiphp: Slot [17] registered [ 0.211066] acpiphp: Slot [18] registered [ 0.212029] acpiphp: Slot [19] registered [ 0.212918] acpiphp: Slot [20] registered [ 0.214044] acpiphp: Slot [21] registered [ 0.215059] acpiphp: Slot [22] registered [ 0.217064] acpiphp: Slot [23] registered [ 0.217935] acpiphp: Slot [24] registered [ 0.218043] acpiphp: Slot [25] registered [ 0.218867] acpiphp: Slot [26] registered [ 0.220086] acpiphp: Slot [27] registered [ 0.220840] acpiphp: Slot [28] registered [ 0.221068] acpiphp: Slot [29] registered [ 0.222043] acpiphp: Slot [30] registered [ 0.222852] acpiphp: Slot [31] registered [ 0.224047] PCI host bridge to bus 0000:00 [ 0.225009] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.226011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.228010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.229000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.229000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.231015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.232160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.234996] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.239183] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.247010] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.251544] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.256017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.259016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.262017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.264484] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266581] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.268031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271803] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.275882] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.286014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.289010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293648] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.307019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.313017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.346016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.369083] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.391024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.402016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.442021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476095] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.477492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.478341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.479000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.479000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.479000] iommu: Default domain type: Passthrough [ 0.479000] SCSI subsystem initialized [ 0.479000] ACPI: bus type USB registered [ 0.479000] usbcore: registered new interface driver usbfs [ 0.479000] usbcore: registered new interface driver hub [ 0.479000] usbcore: registered new device driver usb [ 0.479000] pps_core: LinuxPPS API ver. 1 registered [ 0.479000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.479061] PTP clock support registered [ 0.482023] EDAC MC: Ver: 3.0.0 [ 0.483432] PCI: Using ACPI for IRQ routing [ 0.484752] NetLabel: Initializing [ 0.485012] NetLabel: domain hash size = 128 [ 0.486008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.487201] NetLabel: unlabeled traffic allowed by default [ 0.489009] vgaarb: loaded [ 0.490385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.501023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507836] clocksource: Switched to clocksource kvm-clock [ 0.635932] VFS: Disk quotas dquot_6.6.0 [ 0.637088] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.639159] *** VALIDATE ramfs *** [ 0.640057] *** VALIDATE hugetlbfs *** [ 0.641170] pnp: PnP ACPI init [ 0.643460] pnp: PnP ACPI: found 6 devices [ 0.658029] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.660400] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.661786] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.663134] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.664573] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.666070] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.667860] NET: Registered protocol family 2 [ 0.670490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.674242] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.676720] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.680416] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.682837] TCP: Hash tables configured (established 65536 bind 65536) [ 0.684752] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.686835] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.688913] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690935] NET: Registered protocol family 1 [ 0.693041] RPC: Registered named UNIX socket transport module. [ 0.694751] RPC: Registered udp transport module. [ 0.696061] RPC: Registered tcp transport module. [ 0.697219] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.698846] NET: Registered protocol family 44 [ 0.700085] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.701551] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.703136] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.704727] PCI: CLS 0 bytes, default 64 [ 0.706638] Unpacking initramfs... [ 2.299827] debug: unmapping init [mem 0xffff9a433cc64000-0xffff9a433ffcffff] [ 2.303691] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.305960] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.308678] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.938022] Initialise system trusted keyrings [ 2.940094] Key type blacklist registered [ 2.942046] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.951306] zbud: loaded [ 2.955775] *** VALIDATE nfs *** [ 2.956784] *** VALIDATE nfs4 *** [ 2.957885] pstore: using deflate compression [ 2.960755] Platform Keyring initialized [ 3.066692] NET: Registered protocol family 38 [ 3.068758] Key type asymmetric registered [ 3.074109] Asymmetric key parser 'x509' registered [ 3.076350] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.080429] io scheduler mq-deadline registered [ 3.082567] io scheduler kyber registered [ 3.084326] io scheduler bfq registered [ 3.087603] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.094880] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.097990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100327] ACPI: Power Button [PWRF] [ 3.105512] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.111675] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.124649] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.156227] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.184069] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.206777] Non-volatile memory driver v1.3 [ 3.208434] Linux agpgart interface v0.103 [ 3.244421] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.246799] vda: detected capacity change from 0 to 68624384 [ 3.275072] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.278776] vdb: detected capacity change from 0 to 1073741824 [ 3.290524] libphy: Fixed MDIO Bus: probed [ 3.298407] usbcore: registered new interface driver usbserial_generic [ 3.300438] usbserial: USB Serial support registered for generic [ 3.302815] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.308036] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.309948] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.312399] mousedev: PS/2 mouse device common for all mice [ 3.314854] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.316331] rtc_cmos 00:05: RTC can wake from S4 [ 3.320887] rtc_cmos 00:05: registered as rtc0 [ 3.322428] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.324391] intel_pstate: CPU model not supported [ 3.327140] hid: raw HID events driver (C) Jiri Kosina [ 3.330181] usbcore: registered new interface driver usbhid [ 3.331793] usbhid: USB HID core driver [ 3.333393] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.337029] drop_monitor: Initializing network drop monitor service [ 3.340396] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.345426] Initializing XFRM netlink socket [ 3.347094] NET: Registered protocol family 10 [ 3.349492] Segment Routing with IPv6 [ 3.350336] NET: Registered protocol family 17 [ 3.351572] mpls_gso: MPLS GSO support [ 3.356877] RAS: Correctable Errors collector initialized. [ 3.359783] AVX version of gcm_enc/dec engaged. [ 3.361096] AES CTR mode by8 optimization enabled [ 3.458589] sched_clock: Marking stable (3458453432, 0)->(4303970761, -845517329) [ 3.462537] registered taskstats version 1 [ 3.464456] Loading compiled-in X.509 certificates [ 3.466891] zswap: loaded using pool lzo/zbud [ 3.503143] Key type big_key registered [ 3.516397] Key type encrypted registered [ 3.518431] ima: No TPM chip found, activating TPM-bypass! [ 3.520795] ima: Allocated hash algorithm: sha1 [ 3.522721] ima: No architecture policies found [ 3.524827] evm: Initialising EVM extended attributes: [ 3.526828] evm: security.selinux [ 3.528429] evm: security.ima [ 3.529552] evm: security.capability [ 3.530986] evm: HMAC attrs: 0x1 [ 3.533726] rtc_cmos 00:05: setting system clock to 2026-01-02 00:45:56 UTC (1767314756) [ 3.541736] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 3.545347] debug: unmapping init [mem 0xffffffff8eb82000-0xffffffff8ee58fff] [ 3.557165] Write protecting the kernel read-only data: 28672k [ 3.565431] debug: unmapping init [mem 0xffffffff8d203000-0xffffffff8d3fffff] [ 3.571635] debug: unmapping init [mem 0xffffffff8db14000-0xffffffff8dbfffff] [ 3.626656] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.636518] systemd[1]: Detected virtualization kvm. [ 3.638873] systemd[1]: Detected architecture x86-64. [ 3.640910] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.673238] systemd[1]: No hostname configured. [ 3.675644] systemd[1]: Set hostname to . [ 3.678225] random: systemd: uninitialized urandom read (16 bytes read) [ 3.681837] systemd[1]: Initializing machine ID from random generator. [ 3.789036] random: ln: uninitialized urandom read (6 bytes read) [ 3.963819] random: systemd: uninitialized urandom read (16 bytes read) [ 3.966337] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.971872] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.980782] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.929358] device-mapper: uevent: version 1.0.3 [ 4.931861] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.365427] virtio_net virtio0 ens2: renamed from eth0 [ 6.432805] random: fast init done [ 7.346714] scsi host0: ata_piix [ 7.376466] scsi host1: ata_piix [ 7.380154] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.386210] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.043050] random: crng init done [ 12.044979] random: 7 urandom warning(s) missed due to ratelimiting [ 12.981604] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.931744] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.956848] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.562430] SELinux: Disabled at runtime. [ 17.641600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.655269] systemd[1]: Detected virtualization kvm. [ 17.657673] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.005793] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.014521] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.029366] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.036218] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.043248] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.054341] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.064936] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target.[ 19.228750] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.217532] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.049430] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.072566] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.364708] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.509417] EDAC sbridge: Ver: 1.1.2 [ 23.973934] Key type dns_resolver registered [ 24.560658] NFS: Registering the id_resolver key type [ 24.565910] Key type id_resolver registered [ 24.568215] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-client login: [ 83.745150] libcfs: loading out-of-tree module taints kernel. [ 83.879598] Key type ._llcrypt registered [ 83.881533] Key type .llcrypt registered [ 84.855042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 84.883073] alg: No test for adler32 (adler32-zlib) [ 86.826816] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 87.764110] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 89.519161] Key type lgssc registered [ 92.037552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 164.947423] hrtimer: interrupt took 3404517 ns [ 305.983104] Lustre: Mounted lustre-client [ 313.347351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 331.748162] Lustre: lustre-OST0000-osc-ffff9a43896eb800: disconnect after 22s idle [ 332.030713] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing check_logdir /tmp/testlogs/ [ 338.098068] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing yml_node [ 348.241183] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 352.958457] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 358.116468] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 360.815581] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jan 1 19:51:51 EST 2026 [ 385.898306] LustreError: 9723:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a43896eb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 385.911119] LustreError: 9723:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 385.977892] Lustre: Unmounted lustre-client [ 412.500308] Key type lgssc unregistered [ 412.767740] LNet: 10280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.794987] LNetError: 10280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 412.825716] LNet: Removed LNI 192.168.204.54@tcp [ 413.636506] Key type .llcrypt unregistered [ 413.638224] Key type ._llcrypt unregistered [ 449.510291] Key type ._llcrypt registered [ 449.519502] Key type .llcrypt registered [ 449.736894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.756425] alg: No test for adler32 (adler32-zlib) [ 450.696711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 450.755110] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 450.774621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 455.027640] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 455.162341] LNet: 11267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.176031] LNetError: 11267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.198691] LNet: Removed LNI 192.168.204.54@tcp [ 455.743443] Key type .llcrypt unregistered [ 455.745738] Key type ._llcrypt unregistered [ 462.516569] Key type ._llcrypt registered [ 462.523149] Key type .llcrypt registered [ 464.051131] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:53:35 (1767315215) [ 464.688403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 464.707579] alg: No test for adler32 (adler32-zlib) [ 465.749132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.836516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.856828] LNet: 11682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.895418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.972730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.003292] LNet: 11689:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 473.795420] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:53:45 (1767315225) [ 474.070143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.089786] LNet: 12267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.112697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.171601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 481.684141] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:53:53 (1767315233) [ 481.881416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.929165] LNet: 12850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.979105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.125763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 489.488400] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:54:00 (1767315240) [ 489.724408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.762748] LNet: 13438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.791787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.884104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 497.095722] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:54:08 (1767315248) [ 497.316659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.360429] LNet: 14021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.415250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.505688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 498.136577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 505.292931] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:54:16 (1767315256) [ 505.433723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.467895] LNet: 14606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.497869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.642657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 506.232395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 506.818810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 514.346777] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:54:25 (1767315265) [ 514.503972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.581341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.699810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 522.525203] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:54:33 (1767315273) [ 522.720333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.785657] LNet: 15792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.803204] LNet: 15792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 522.849892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.942661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 522.968684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 523.704269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 523.763098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 524.399971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 524.451377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 525.049650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 525.089356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 525.773436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 525.845744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 526.433874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 526.497564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 527.229862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 527.283692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 528.070456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 528.105971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 528.732983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 528.777229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 529.619277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 529.667453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 530.445564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 530.536276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 531.431166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 531.518757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 532.355402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 532.422349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 541.274673] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:54:52 (1767315292) [ 541.415961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.480976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.590646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 541.620789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 549.619575] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:55:00 (1767315300) [ 549.887025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.966763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.048458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 550.086639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 557.757858] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:55:09 (1767315309) [ 558.027813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.050599] LNet: 17649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.054102] LNet: 17649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 558.080931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.238421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 558.283351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 566.117252] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:55:17 (1767315317) [ 566.233397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.337267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.444252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 566.508326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 574.584335] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:55:26 (1767315326) [ 574.714770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.796732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.912718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 574.972919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 582.068458] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:55:33 (1767315333) [ 582.238712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.274919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.377463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 582.434074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 589.683176] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:55:40 (1767315340) [ 589.851366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.960343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.087091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 590.706509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 597.981984] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:55:49 (1767315349) [ 598.086037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.130301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.403984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.663867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 606.122806] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:55:57 (1767315357) [ 606.434300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.534892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.745848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 607.175443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 615.787301] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:56:06 (1767315366) [ 615.911856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.014994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.869641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.386210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 625.129170] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:56:16 (1767315376) [ 625.370979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.393596] LNet: 22482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.400973] LNet: 22482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 625.431660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.501361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.547228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 633.615750] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:56:24 (1767315384) [ 633.779134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.800835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.841661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.880996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 641.882832] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:56:33 (1767315393) [ 641.985462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.034077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.160389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 649.211617] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:56:40 (1767315400) [ 649.441446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.567935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.659225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 657.698766] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:56:48 (1767315408) [ 657.855574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.936203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.017725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 658.690694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 666.024131] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:56:57 (1767315417) [ 666.265880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.318778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.451856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 666.490158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 675.675690] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:57:06 (1767315426) [ 675.824424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.899064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.041326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 676.122482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 684.130388] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:57:15 (1767315435) [ 684.255685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.337939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.467691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 684.525735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 692.599628] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:57:23 (1767315443) [ 692.693794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.742873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.895473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 692.970628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 700.123912] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:57:31 (1767315451) [ 700.315886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.365890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.456432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 700.514541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 700.537268] LNetError: 27798:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 700.575978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 700.603912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 707.851845] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:57:39 (1767315459) [ 708.104301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.130378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.193628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OZ4Ffca3os [ 708.268356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OZ4Ffca3os [ 708.323968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OZ4Ffca3os [ 715.125944] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:57:46 (1767315466) [ 715.475275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.549726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.636656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 715.695316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 715.776495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 722.392791] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:57:53 (1767315473) [ 722.523644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.607824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.739381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 726.426627] LNet: Message forwarding will be enabled in 2s [ 729.503139] LNet: Message forwarding enabled [ 730.038434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 730.074214] LNet: Message forwarding will stop in 2s to 6s [ 734.598960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 735.323897] LNet: Message forwarding disabled [ 737.433041] LNet: Message forwarding will be enabled in 2s [ 741.475070] LNet: Message forwarding enabled [ 744.907362] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:58:16 (1767315496) [ 745.111817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.584992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.653381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 753.770309] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:58:25 (1767315505) [ 753.902950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.927728] LNet: 30823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.934334] LNet: 30823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 753.971619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.075173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 754.109876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 754.150457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 754.190667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 754.226782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 754.260977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 754.290815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 754.320833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 754.359857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 754.399492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 754.438879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 754.470020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 754.498906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 754.526228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 754.571427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 754.604870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 754.641214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 754.681449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 754.715956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 754.741439] LNetError: 30870:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 760.840364] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:58:32 (1767315512) [ 761.136514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.259458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.389389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 768.644600] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:58:40 (1767315520) [ 770.581518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg454-server [ 772.516384] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:58:43 (1767315523) [ 772.619286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.680823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.716903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.756079] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 774.494173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[55-59]@tcp [ 774.517858] LNet: 32588:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 774.582550] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 774.614056] LNetError: 32375:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.56@tcp has gone from up to down [ 775.586453] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.57@tcp is being used as a gateway but routing feature is not turned on [ 775.596851] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 776.611483] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 776.630831] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 778.664124] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.57@tcp is being used as a gateway but routing feature is not turned on [ 778.683506] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 781.503895] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:58:52 (1767315532) [ 782.769520] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.57@tcp is being used as a gateway but routing feature is not turned on [ 782.791636] LNetError: 32375:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 783.495222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg454-server [ 785.400106] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:58:56 (1767315536) [ 785.560844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.586052] LNetError: 33517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.688561] LNet: Removed LNI 192.168.204.54@tcp [ 786.734611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.805647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.823779] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 787.581840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[55-60/2]@tcp [ 787.605674] LNet: 33640:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 787.621031] LNet: 33640:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 787.635842] LNetError: 33527:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.57@tcp has gone from up to down [ 787.650855] LNetError: 33527:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 787.659207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[55-60/2]@tcp [ 793.935484] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:59:05 (1767315545) [ 794.070441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.126854] LNetError: 34227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.163738] LNet: Removed LNI 192.168.204.54@tcp [ 794.190834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.289697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 794.336444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.366807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.454717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 794.526957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.594612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.667655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 794.826636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.918385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.048996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 795.156274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.217714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.274947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 795.365346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.411198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.479385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 795.562820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.605375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.654413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 795.751480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.812594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.898588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 795.983676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.038280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.113701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 796.223773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.290440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.388060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 796.548793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.609557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.682625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 796.756961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.802244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.928927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 803.906212] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:59:15 (1767315555) [ 806.103882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg454-server [ 808.732602] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:59:19 (1767315559) [ 810.964389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg454-server [ 813.565895] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:59:24 (1767315564) [ 815.697644] Key type .llcrypt unregistered [ 815.699160] Key type ._llcrypt unregistered [ 827.176888] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 827.415668] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.446903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 827.463342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.581972] Key type ._llcrypt registered [ 830.587414] Key type .llcrypt registered [ 830.714784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.731952] alg: No test for adler32 (adler32-zlib) [ 831.703493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.761705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.781622] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 832.591894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 832.609746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.067085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 835.107853] LNet: Removed LNI 10.1.2.3@tcp [ 835.283140] LNet: 36353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.288519] LNetError: 36353:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.306146] LNet: Removed LNI 192.168.204.54@tcp [ 836.087851] Key type .llcrypt unregistered [ 836.095729] Key type ._llcrypt unregistered [ 846.938783] Key type ._llcrypt registered [ 846.941410] Key type .llcrypt registered [ 848.376919] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:59:59 (1767315599) [ 849.346510] Key type .llcrypt unregistered [ 849.348939] Key type ._llcrypt unregistered [ 857.274995] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.470178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 857.479215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 857.488119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 859.960401] Key type ._llcrypt registered [ 859.961918] Key type .llcrypt registered [ 860.145429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.165313] alg: No test for adler32 (adler32-zlib) [ 861.160921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.222865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.249181] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 862.229912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 862.249542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.496454] LNet: 37843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.513634] LNetError: 37843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.521166] LNet: Removed LNI 192.168.204.54@tcp [ 866.543079] LNet: Removed LNI 10.1.2.3@tcp [ 867.419934] Key type .llcrypt unregistered [ 867.422208] Key type ._llcrypt unregistered [ 886.551698] Key type ._llcrypt registered [ 886.563872] Key type .llcrypt registered [ 889.053112] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:00:39 (1767315639) [ 889.782411] Key type .llcrypt unregistered [ 889.784977] Key type ._llcrypt unregistered [ 898.262397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.460155] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 898.483488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 898.501350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 901.435347] Key type ._llcrypt registered [ 901.436703] Key type .llcrypt registered [ 901.840257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.855552] alg: No test for adler32 (adler32-zlib) [ 902.948686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.085411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 903.163359] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 905.671749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 905.692079] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.834403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 907.881381] LNet: Removed LNI 10.1.2.3@tcp [ 907.908895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 907.993910] LNet: Removed LNI 10.1.2.31@tcp [ 908.025071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 908.051147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.314360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 909.365698] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 910.587658] LNet: 39775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.598475] LNetError: 39775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.607780] LNet: Removed LNI 10.1.2.3@tcp [ 910.620359] LNet: Removed LNI 10.1.2.31@tcp [ 911.586518] Key type .llcrypt unregistered [ 911.592337] Key type ._llcrypt unregistered [ 924.589664] Key type ._llcrypt registered [ 924.592448] Key type .llcrypt registered [ 925.623334] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:01:17 (1767315677) [ 926.120997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.133450] alg: No test for adler32 (adler32-zlib) [ 927.098623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.175234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.211780] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 928.083660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.111350] LNet: 40725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.117854] LNetError: 40725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.129773] LNet: Removed LNI 192.168.204.54@tcp [ 928.146808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.193928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.54@tcp [ 928.229331] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 929.108610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.54@tcp [ 929.140841] LNet: Removed LNI 192.168.204.54@tcp [ 929.248677] LNet: 40860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.894387] Key type .llcrypt unregistered [ 929.895327] Key type ._llcrypt unregistered [ 942.157401] Key type ._llcrypt registered [ 942.158929] Key type .llcrypt registered [ 943.474685] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:01:35 (1767315695) [ 945.332168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg454-server [ 948.033785] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:01:38 (1767315698) [ 948.701507] Key type .llcrypt unregistered [ 948.708952] Key type ._llcrypt unregistered [ 955.685697] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 955.795869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 955.798122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 955.800708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 958.530863] Key type ._llcrypt registered [ 958.533327] Key type .llcrypt registered [ 958.883509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.894720] alg: No test for adler32 (adler32-zlib) [ 959.856588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.000602] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 961.680734] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 964.451958] LNet: 42686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.455578] LNetError: 42686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.461680] LNet: Removed LNI 192.168.204.54@tcp1 [ 964.476634] LNet: Removed LNI 10.1.2.3@tcp2 [ 965.418392] Key type .llcrypt unregistered [ 965.419800] Key type ._llcrypt unregistered [ 977.522341] Key type ._llcrypt registered [ 977.524469] Key type .llcrypt registered [ 978.863363] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:02:10 (1767315730) [ 979.423512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.437096] alg: No test for adler32 (adler32-zlib) [ 980.344662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.389878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 980.408157] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 981.612791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.632851] LNet: 43651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.639225] LNetError: 43651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.656398] LNet: Removed LNI 192.168.204.54@tcp [ 981.708683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.761531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 981.784164] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 988.899856] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:02:20 (1767315740) [ 989.147343] LNet: 44242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.172542] LNetError: 44242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.204352] LNet: Removed LNI 192.168.204.54@tcp [ 990.240530] Key type .llcrypt unregistered [ 990.242562] Key type ._llcrypt unregistered [ 995.764658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 995.914684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 995.940447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 995.959367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 998.943969] Key type ._llcrypt registered [ 998.949375] Key type .llcrypt registered [ 999.163642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.198119] alg: No test for adler32 (adler32-zlib) [ 1000.052349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.083900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.096822] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1000.682804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1000.697755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.361712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.374240] LNet: 44982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.379631] LNetError: 44982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.383513] LNet: Removed LNI 192.168.204.54@tcp [ 1002.391144] LNet: Removed LNI 10.1.2.3@tcp [ 1002.432290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.471718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1002.508689] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1002.519774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.080278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.092049] LNet: 45010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.094929] LNetError: 45010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.098124] LNetError: 45010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1003.105983] LNet: Removed LNI 192.168.204.54@tcp [ 1003.117948] LNet: Removed LNI 10.1.2.3@tcp [ 1003.143465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.206843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1003.228589] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1003.233596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.453518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1003.473305] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 1004.782779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.824889] LNet: 45172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.853425] LNetError: 45172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.861823] LNetError: 45172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1004.868824] LNet: Removed LNI 192.168.204.54@tcp [ 1004.870808] LNet: Removed LNI 10.1.2.3@tcp [ 1004.888228] LNet: Removed LNI 192.168.204.54@tcp2 [ 1004.915516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.060614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1005.101348] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1005.113750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.125316] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 1015.489644] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:02:46 (1767315766) [ 1015.576138] LNet: 45761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.588573] LNetError: 45761:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.591959] LNetError: 45761:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1015.595655] LNet: Removed LNI 192.168.204.54@tcp [ 1015.597950] LNet: Removed LNI 10.1.2.3@tcp [ 1015.624267] LNet: Removed LNI 192.168.204.54@tcp2 [ 1016.826771] Key type .llcrypt unregistered [ 1016.830493] Key type ._llcrypt unregistered [ 1023.589610] Key type ._llcrypt registered [ 1023.591707] Key type .llcrypt registered [ 1023.856267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.883067] alg: No test for adler32 (adler32-zlib) [ 1024.786994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.898285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.957400] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1026.439247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.483470] LNet: 46161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.490900] LNetError: 46161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.522959] LNet: Removed LNI 192.168.204.54@tcp [ 1026.551554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.695510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1026.764498] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1027.380794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.410759] LNet: 46191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.416198] LNetError: 46191:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.430749] LNet: Removed LNI 192.168.204.54@tcp [ 1027.444793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.494457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1027.515493] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1028.126553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.171178] LNet: Removed LNI 192.168.204.54@tcp [ 1028.220226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.317762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1028.341467] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1028.706799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1028.918537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1028.921033] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1028.923968] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1031.472114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.507710] LNet: 46301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.517479] LNet: 46301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1031.534984] LNetError: 46301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.543588] LNetError: 46301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1031.555881] LNet: Removed LNI 192.168.204.54@tcp [ 1031.586866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.742706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1031.797285] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1041.190509] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:03:12 (1767315792) [ 1041.522582] LNet: 46899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.530838] LNetError: 46899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.550531] LNet: Removed LNI 192.168.204.54@tcp [ 1042.169406] Key type .llcrypt unregistered [ 1042.172184] Key type ._llcrypt unregistered [ 1049.803268] Key type ._llcrypt registered [ 1049.809158] Key type .llcrypt registered [ 1050.130509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.176156] alg: No test for adler32 (adler32-zlib) [ 1051.304733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.421789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.521855] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1052.503460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.543561] LNet: 47294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.562662] LNetError: 47294:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.576208] LNet: Removed LNI 192.168.204.54@tcp [ 1052.612418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.739153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1052.784972] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1061.208531] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:03:32 (1767315812) [ 1061.511792] LNet: 47887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.526227] LNetError: 47887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.556206] LNet: Removed LNI 192.168.204.54@tcp [ 1062.304201] Key type .llcrypt unregistered [ 1062.305994] Key type ._llcrypt unregistered [ 1069.931892] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1070.154486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1070.164179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1070.172262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1073.008332] Key type ._llcrypt registered [ 1073.011457] Key type .llcrypt registered [ 1073.333996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.369793] alg: No test for adler32 (adler32-zlib) [ 1074.549331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.623244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1074.667444] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1076.129954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1076.166874] LNet: 48329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.177441] LNetError: 48329:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.217290] LNet: Removed LNI 192.168.204.54@tcp [ 1076.260621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.423337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1076.454736] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1085.385216] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:03:56 (1767315836) [ 1085.662838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.723515] LNet: 48924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.738694] LNetError: 48924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.802769] LNet: Removed LNI 192.168.204.54@tcp [ 1085.854373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.969368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1093.127210] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:04:04 (1767315844) [ 1093.635389] LNet: 49510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.351544] Key type .llcrypt unregistered [ 1094.353422] Key type ._llcrypt unregistered [ 1101.762668] Key type ._llcrypt registered [ 1101.766881] Key type .llcrypt registered [ 1102.052365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1102.072938] alg: No test for adler32 (adler32-zlib) [ 1103.078103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.145672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1103.166041] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1104.454577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.467964] LNet: 49904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.471968] LNetError: 49904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.489277] LNet: Removed LNI 192.168.204.54@tcp [ 1104.530447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.646695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1104.696735] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1112.409251] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:04:23 (1767315863) [ 1112.554076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.611784] LNet: 50491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.645755] LNetError: 50491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.672699] LNet: Removed LNI 192.168.204.54@tcp [ 1112.751297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.849339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1120.503720] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:04:31 (1767315871) [ 1122.322847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1124.390950] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:04:35 (1767315875) [ 1126.550174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1128.487334] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:04:39 (1767315879) [ 1130.396983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1131.957919] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:04:43 (1767315883) [ 1133.705975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1135.627942] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:04:47 (1767315887) [ 1137.396573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1139.082236] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:04:50 (1767315890) [ 1139.246229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1139.289183] LNet: 52553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.314774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.360804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1139.381198] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1140.162958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.54 [ 1140.184561] LNet: 52686:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1140.195780] LNet: Removed LNI 192.168.204.54@tcp [ 1146.654902] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:04:58 (1767315898) [ 1146.959531] LNet: 53257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.964954] LNetError: 53257:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.980853] LNet: Removed LNI 192.168.204.54@tcp [ 1147.589352] Key type .llcrypt unregistered [ 1147.590809] Key type ._llcrypt unregistered [ 1154.813529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1155.130705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1155.137951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1155.144719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1157.816801] Key type ._llcrypt registered [ 1157.819835] Key type .llcrypt registered [ 1158.007596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.016977] alg: No test for adler32 (adler32-zlib) [ 1159.449451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1167.008578] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:05:18 (1767315918) [ 1167.229952] LNet: 54154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.245231] LNetError: 54154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1167.285414] LNet: Removed LNI 10.1.2.3@tcp [ 1168.170961] Key type .llcrypt unregistered [ 1168.172065] Key type ._llcrypt unregistered [ 1175.494245] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1175.771586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1175.778688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1175.786788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1177.509851] Key type ._llcrypt registered [ 1177.514434] Key type .llcrypt registered [ 1177.758794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.782768] alg: No test for adler32 (adler32-zlib) [ 1178.845607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1185.917892] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:05:37 (1767315937) [ 1186.103541] LNet: 55032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.109780] LNetError: 55032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1186.134685] LNet: Removed LNI 10.1.2.3@tcp [ 1186.907510] Key type .llcrypt unregistered [ 1186.910286] Key type ._llcrypt unregistered [ 1193.964081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1194.228842] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1194.231708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1194.234652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1196.991689] Key type ._llcrypt registered [ 1196.993531] Key type .llcrypt registered [ 1197.173651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1197.213971] alg: No test for adler32 (adler32-zlib) [ 1198.200974] LNet: 55337:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1198.211953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1205.831989] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:05:56 (1767315956) [ 1206.038504] LNet: 55925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.049480] LNetError: 55925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1206.063793] LNet: Removed LNI 10.1.2.3@tcp [ 1206.847967] Key type .llcrypt unregistered [ 1206.850173] Key type ._llcrypt unregistered [ 1213.322452] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1213.608034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1213.610990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1213.616700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1216.363873] Key type ._llcrypt registered [ 1216.366260] Key type .llcrypt registered [ 1216.548210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.565032] alg: No test for adler32 (adler32-zlib) [ 1217.438927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.476092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1227.634964] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:06:19 (1767315979) [ 1228.279382] LNet: 57154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.285443] LNetError: 57154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1228.297839] LNet: Removed LNI 10.1.2.3@tcp [ 1229.205706] Key type .llcrypt unregistered [ 1229.208629] Key type ._llcrypt unregistered [ 1238.066811] Key type ._llcrypt registered [ 1238.070133] Key type .llcrypt registered [ 1238.319642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.338347] alg: No test for adler32 (adler32-zlib) [ 1239.350322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1239.385542] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1247.714321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1248.545328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1248.586798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1248.619570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1248.630455] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1248.869449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.915109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1248.965481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.010294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.036637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.067794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.102325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.134621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.161367] LNet: 57588:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.189650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.223766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.276807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.341235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.396551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.442149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.483562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.142291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1250.174376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1250.210210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1250.244994] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1250.624392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.655131] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1250.703321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1250.762771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.848637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.895546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.935690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.978848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.038486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.082589] LNet: 57697:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.107434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.194802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.269246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.297293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.357935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.395765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.427720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.082957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1252.156739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1252.208361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1252.224953] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1252.541235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.588986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1252.637871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.698470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.749572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.808562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.844767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.925926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.945820] LNet: 57807:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.989211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.048929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.088615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.141317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.177904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.210327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.243785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.917632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1253.970633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1254.012694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1254.319178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.378474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1254.420433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.465877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.492712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.518789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.562888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.627518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.665300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.705300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.749283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.801406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.856351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.906047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.975960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.514514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1255.580795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1255.624276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1255.640927] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1255.646472] LNet: Skipped 1 previous similar message [ 1255.856339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.880443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1255.906881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.931149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.953699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.975665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.011876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.071024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.108910] LNet: 58025:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.120559] LNet: 58025:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1256.144777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.173156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.200389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.221115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.250789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.285935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.311319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.907825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1256.943563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1256.977502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1257.261938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.338717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1257.382907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.424947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.496726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.533493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.564716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.582696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.621916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.656599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.696566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.750482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.793701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.836697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.872891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1258.160767] LNet: 58174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.176495] LNetError: 58174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.203949] LNet: Removed LNI 192.168.204.54@tcp [ 1258.738798] Key type .llcrypt unregistered [ 1258.741666] Key type ._llcrypt unregistered [ 1276.354487] Key type ._llcrypt registered [ 1276.356280] Key type .llcrypt registered [ 1277.373439] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:07:09 (1767316029) [ 1278.260368] Key type .llcrypt unregistered [ 1278.262508] Key type ._llcrypt unregistered [ 1285.155419] Key type ._llcrypt registered [ 1285.158416] Key type .llcrypt registered [ 1285.396467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.424866] alg: No test for adler32 (adler32-zlib) [ 1286.398501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.422112] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1294.256740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1304.924512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.974429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.022135] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1306.511895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1306.554332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 1 [ 1306.598492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1306.638585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 1 [ 1306.692578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_interrupt -r 1 [ 1306.756483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 1 [ 1306.812445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_interrupt -r 1 [ 1306.897092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 1 [ 1306.963462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1307.021308] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1307.028635] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1307.322238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.357540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1307.391950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.425500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.470970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.526541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.598842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.664159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.682495] LNet: 59637:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.726950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.792172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.826324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.871567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.899640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.957606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.997270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.672322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1308.703756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 1 [ 1308.747820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1308.781302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 1 [ 1308.813770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_dropped -r 1 [ 1308.848335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 1 [ 1308.881533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_dropped -r 1 [ 1308.918920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 1 [ 1308.951699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1308.970854] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1308.975381] LNet: Skipped 2 previous similar messages [ 1309.201554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.235129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1309.260175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.292318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.318421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.352043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.403890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.453368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.486983] LNet: 59758:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.522330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1309.564987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1309.623740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.738884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.781252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.830671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.866890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1310.631796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1310.694554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 1 [ 1310.773969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1310.839746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 1 [ 1310.921731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_aborted -r 1 [ 1310.984544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 1 [ 1311.071507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_aborted -r 1 [ 1311.131812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 1 [ 1311.192664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1311.232083] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1311.242539] LNet: Skipped 2 previous similar messages [ 1311.511503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.548281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1311.579398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.607229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.643230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.676278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.711923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.737038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.753756] LNet: 59880:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.773511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1311.809243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1311.852908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1311.895199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.934706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1311.974249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.002370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1312.640512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1312.689263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 1 [ 1312.741295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1312.789448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 1 [ 1312.817609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_no_route -r 1 [ 1312.849564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 1 [ 1312.890873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_no_route -r 1 [ 1312.938114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 1 [ 1312.986781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1313.318696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1313.349220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1313.379814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1313.407985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1313.443320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.477437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.515511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.573249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.663664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.712498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.772699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.808364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.875081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.937673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1314.010766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1314.741388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1314.780571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 1 [ 1314.830251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1314.858304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 1 [ 1314.894076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_timeout -r 1 [ 1314.919700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 1 [ 1314.956272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_timeout -r 1 [ 1314.993747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 1 [ 1315.036716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1315.064952] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1315.076677] LNet: Skipped 5 previous similar messages [ 1315.346701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1315.375965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1315.407985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.435231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.456192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.470766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.490817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.519245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.540288] LNet: 60123:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.546564] LNet: 60123:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1315.577960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.624909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.658749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.678783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.700829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.725669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.765650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.200231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1316.269385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_error -r 1 [ 1316.327093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1316.373148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_error -r 1 [ 1316.430986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_error -r 1 [ 1316.470958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_error -r 1 [ 1316.518346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_error -r 1 [ 1316.594390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_error -r 1 [ 1316.656664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1316.962304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.004185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1317.056118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.099521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.149810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.206687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.304399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.360144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.413604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1317.473221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1317.526630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.567713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.635652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.674310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.728223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1323.482020] LNet: 60319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.492105] LNetError: 60319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.503208] LNet: Removed LNI 192.168.204.54@tcp [ 1324.519554] LNet: Removed LNI 192.168.204.54@tcp1 [ 1325.106767] Key type .llcrypt unregistered [ 1325.108486] Key type ._llcrypt unregistered [ 1341.357988] Key type ._llcrypt registered [ 1341.361908] Key type .llcrypt registered [ 1342.615201] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:08:14 (1767316094) [ 1343.147616] Key type .llcrypt unregistered [ 1343.150176] Key type ._llcrypt unregistered [ 1348.999816] Key type ._llcrypt registered [ 1349.006589] Key type .llcrypt registered [ 1349.209675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.224425] alg: No test for adler32 (adler32-zlib) [ 1350.108743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1350.128811] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1357.287170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1357.878175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1357.903542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1357.929272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1357.941963] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1358.198136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.226239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1358.253225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.292968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.330514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.365340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.412882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.441426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.456401] LNet: 61565:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.473629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.497405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.526274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.552084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.592560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.622649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.647824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.147565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1359.205595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1359.254492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1359.282481] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1359.493123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.522432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1359.549549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.574168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.605641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.645988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.685833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.752474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.767746] LNet: 61674:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.780745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.813833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.842266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.879661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.941229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.004408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.039439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.587354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1360.629872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1360.660328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1360.678850] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1360.958278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.004183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1361.041354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.070183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.104504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.141905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.175480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1361.208298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1361.239568] LNet: 61783:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1361.264263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1361.303795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1361.332346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.384275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1361.412344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.436841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.476351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.709779] LNet: 61824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.716236] LNetError: 61824:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.731402] LNet: Removed LNI 192.168.204.54@tcp [ 1361.984417] Key type .llcrypt unregistered [ 1361.987972] Key type ._llcrypt unregistered [ 1379.948696] Key type ._llcrypt registered [ 1379.949844] Key type .llcrypt registered [ 1381.135229] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:08:52 (1767316132) [ 1381.964778] Key type .llcrypt unregistered [ 1381.968299] Key type ._llcrypt unregistered [ 1388.790161] Key type ._llcrypt registered [ 1388.792806] Key type .llcrypt registered [ 1389.038305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1389.067312] alg: No test for adler32 (adler32-zlib) [ 1389.996914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1390.017190] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1397.519278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1407.689294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.761568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.782818] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1409.589888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1409.620476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 1 [ 1409.677564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1409.709851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 1 [ 1409.747040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_dropped -r 1 [ 1409.788392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 1 [ 1409.832926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_dropped -r 1 [ 1409.876415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 1 [ 1409.924838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1409.951585] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1409.955747] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1410.245616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.283797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1410.316726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.349953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.384676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.428917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.471923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.501575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.524409] LNet: 63286:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.550558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1410.594687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1410.625685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.647904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.670861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.701703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1410.732691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1410.753420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.275368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1411.315373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_error -r 1 [ 1411.346418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1411.368568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_error -r 1 [ 1411.406676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_error -r 1 [ 1411.452907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_error -r 1 [ 1411.523125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_error -r 1 [ 1411.583359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_error -r 1 [ 1411.658359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1411.683683] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1411.688654] LNet: Skipped 2 previous similar messages [ 1411.955948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1411.986882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1412.027186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.066409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.102928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1412.136345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1412.186195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1412.246311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1412.281739] LNet: 63409:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1412.299848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1412.337987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1412.384168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.423032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1412.455551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1412.508986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1412.574072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1412.638056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1413.217264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1413.263995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_timeout -r 1 [ 1413.293262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1413.321321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_timeout -r 1 [ 1413.343996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_timeout -r 1 [ 1413.374283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_timeout -r 1 [ 1413.408262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_timeout -r 1 [ 1413.449989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_timeout -r 1 [ 1413.479917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.154@tcp [ 1413.494877] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1413.497495] LNet: Skipped 1 previous similar message [ 1413.747541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1413.786918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1413.822874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1413.876132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1413.925357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1413.989985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.079283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.144777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.178642] LNet: 63533:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.204867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1414.256605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1414.310990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.353281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1414.386699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.424585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1414.458185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1414.487900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.612169] LNet: 63610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.622873] LNetError: 63610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.639123] LNet: Removed LNI 192.168.204.54@tcp [ 1420.643773] LNetError: 63610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.664705] LNet: Removed LNI 192.168.204.54@tcp1 [ 1421.269980] Key type .llcrypt unregistered [ 1421.275265] Key type ._llcrypt unregistered [ 1440.058830] Key type ._llcrypt registered [ 1440.062408] Key type .llcrypt registered [ 1441.692749] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:09:52 (1767316192) [ 1442.458911] Key type .llcrypt unregistered [ 1442.464890] Key type ._llcrypt unregistered [ 1449.379867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1449.579973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1449.582120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1449.594842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1452.502899] Key type ._llcrypt registered [ 1452.505177] Key type .llcrypt registered [ 1452.663940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.681692] alg: No test for adler32 (adler32-zlib) [ 1453.624933] LNet: 64683:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1453.634810] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1453.664652] LNet: 64697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.668754] LNetError: 64697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.682208] LNet: Removed LNI 192.168.204.54@tcp [ 1454.565689] Key type .llcrypt unregistered [ 1454.570749] Key type ._llcrypt unregistered [ 1461.737481] Key type ._llcrypt registered [ 1461.739973] Key type .llcrypt registered [ 1462.014951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1462.048530] alg: No test for adler32 (adler32-zlib) [ 1463.012715] LNet: 64952:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1463.023261] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1463.025780] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1463.049638] LNet: 64966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.054762] LNetError: 64966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.059723] LNet: Removed LNI 192.168.204.54@tcp [ 1463.073357] LNet: Removed LNI 10.1.2.3@tcp1 [ 1463.735625] Key type .llcrypt unregistered [ 1463.737794] Key type ._llcrypt unregistered [ 1470.065885] Key type ._llcrypt registered [ 1470.068186] Key type .llcrypt registered [ 1470.241872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.268662] alg: No test for adler32 (adler32-zlib) [ 1471.323907] LNet: 65222:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1471.343990] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1471.346258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1471.401558] LNet: 65236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.410632] LNetError: 65236:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.415551] LNet: Removed LNI 192.168.204.54@tcp [ 1471.425131] LNet: Removed LNI 10.1.2.3@tcp [ 1472.189538] Key type .llcrypt unregistered [ 1472.193699] Key type ._llcrypt unregistered [ 1478.952561] Key type ._llcrypt registered [ 1478.954987] Key type .llcrypt registered [ 1479.247723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.269678] alg: No test for adler32 (adler32-zlib) [ 1480.252395] LNet: 65493:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1480.267106] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1480.268767] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1480.296569] LNet: 65507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.301720] LNetError: 65507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.306353] LNet: Removed LNI 192.168.204.54@tcp [ 1480.322683] LNet: Removed LNI 10.1.2.3@tcp [ 1481.236838] Key type .llcrypt unregistered [ 1481.239558] Key type ._llcrypt unregistered [ 1488.420344] Key type ._llcrypt registered [ 1488.422880] Key type .llcrypt registered [ 1488.557776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1488.575844] alg: No test for adler32 (adler32-zlib) [ 1489.494722] LNet: 65768:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1489.509177] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1489.511348] LNetError: 65768:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1489.514511] LNetError: Error -100 starting up LNI tcp [ 1489.539975] LNet: Removed LNI 192.168.204.54@tcp [ 1490.274377] Key type .llcrypt unregistered [ 1490.276561] Key type ._llcrypt unregistered [ 1500.817489] Key type ._llcrypt registered [ 1500.819299] Key type .llcrypt registered [ 1502.337939] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:10:53 (1767316253) [ 1503.242389] Key type .llcrypt unregistered [ 1503.246129] Key type ._llcrypt unregistered [ 1510.111559] Key type ._llcrypt registered [ 1510.113641] Key type .llcrypt registered [ 1510.305505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1510.320802] alg: No test for adler32 (adler32-zlib) [ 1511.276984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1511.305033] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1518.434464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1519.123953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1519.159778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1519.195390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1519.213451] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1519.431763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1519.465682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1519.482319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1519.500702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1519.525647] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1519.531703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1519.586700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1519.623775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.660304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1519.674891] LNet: 67020:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1519.691088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1519.723240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1519.750235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.788878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.825264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1519.850675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1519.969576] LNet: 67058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.974374] LNetError: 67058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.998264] LNet: Removed LNI 192.168.204.54@tcp [ 1521.420471] Key type .llcrypt unregistered [ 1521.421814] Key type ._llcrypt unregistered [ 1540.791060] Key type ._llcrypt registered [ 1540.793203] Key type .llcrypt registered [ 1541.099484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1541.125393] alg: No test for adler32 (adler32-zlib) [ 1542.022263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1542.054389] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1549.975688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1561.302252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.369502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1561.393205] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1562.696204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1562.723932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e network_timeout -r 1 [ 1562.741895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1562.758587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e network_timeout -r 1 [ 1562.785974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e network_timeout -r 1 [ 1562.809819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e network_timeout -r 1 [ 1562.832675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e network_timeout -r 1 [ 1562.861957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e network_timeout -r 1 [ 1562.889190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1562.904782] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1563.254718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1563.300929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1563.350780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1563.411916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1563.475834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1563.518360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1563.552316] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1563.554071] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1563.561644] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1563.567467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1563.574956] LNet: Skipped 1 previous similar message [ 1563.617694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1563.644627] LNet: 67934:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1563.665388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1563.696516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1563.716295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1563.734804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1563.750550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1563.800540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1568.876762] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1569.119571] LNet: 68007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.122757] LNetError: 68007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.125800] LNet: Removed LNI 192.168.204.54@tcp [ 1569.136301] LNet: Removed LNI 192.168.204.54@tcp1 [ 1569.546530] Key type .llcrypt unregistered [ 1569.549775] Key type ._llcrypt unregistered [ 1587.980893] Key type ._llcrypt registered [ 1587.983380] Key type .llcrypt registered [ 1589.020601] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:12:20 (1767316340) [ 1589.551497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.567856] alg: No test for adler32 (adler32-zlib) [ 1590.528750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.603649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.642120] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1591.387463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1591.409836] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1592.228919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1592.301671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1592.375864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1592.411724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1592.461462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1592.492118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1592.529489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1592.551481] LNetError: 69105:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1592.611451] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1592.614537] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1594.656148] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1595.021753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1598.764100] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1599.869760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1599.902904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1599.960599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.988170] LNet: 69212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.993795] LNetError: 69212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.005320] LNet: Removed LNI 192.168.204.54@tcp [ 1600.017232] LNet: Removed LNI 192.168.204.54@tcp1 [ 1600.086681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.205423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.254487] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1601.589586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1601.629956] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1603.042946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1603.097339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1603.222182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1603.289877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1603.343451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1603.377462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1603.419895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1603.463674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1603.478949] LNetError: 69485:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1604.255619] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1607.164455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1610.400535] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1610.417205] LNet: Skipped 1 previous similar message [ 1615.439870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1615.497174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1615.565682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.831900] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:12:54 (1767316374) [ 1623.994780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.009616] LNet: 70201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.020703] LNetError: 70201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.025464] LNetError: 70201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1624.033663] LNet: Removed LNI 192.168.204.54@tcp [ 1624.047742] LNet: Removed LNI 192.168.204.54@tcp1 [ 1624.069754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.150919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.187072] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1625.007853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1625.031726] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1625.944338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1626.009877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1626.040865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1626.073883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1626.130852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1626.161767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1626.205027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 0 [ 1626.303242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1627.167348] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.54@tcp [ 1627.169614] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1627.191777] LNet: Skipped 1 previous similar message [ 1641.981820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1642.180644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1642.220762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.271715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 500 [ 1642.403817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.051721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.097628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.113359] LNet: 70695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.131513] LNetError: 70695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.138385] LNetError: 70695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1649.148375] LNet: Removed LNI 192.168.204.54@tcp [ 1649.176815] LNet: Removed LNI 192.168.204.54@tcp1 [ 1649.211819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.305336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.331221] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1650.234501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.252554] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1651.023858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1651.056598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1651.120657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1651.148618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1651.189883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1651.228868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1651.271939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1651.308712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.54@tcp --health 0 [ 1652.319482] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1652.325665] LNet: Skipped 3 previous similar messages [ 1652.721810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1663.358742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1663.400136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1663.438871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1670.463697] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:13:42 (1767316422) [ 1670.924689] LNet: 71743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.933732] LNetError: 71743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.938750] LNetError: 71743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1670.955160] LNet: Removed LNI 192.168.204.54@tcp [ 1670.971286] LNet: Removed LNI 192.168.204.54@tcp1 [ 1671.598857] Key type .llcrypt unregistered [ 1671.610661] Key type ._llcrypt unregistered [ 1678.430863] Key type ._llcrypt registered [ 1678.436774] Key type .llcrypt registered [ 1678.651864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1678.662630] alg: No test for adler32 (adler32-zlib) [ 1679.618631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1679.648505] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1689.127846] Lustre: DEBUG MARKER: Initial discovery [ 1689.372248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1691.977898] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1692.296150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1692.351336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1692.369705] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1692.439615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1692.519243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.154@tcp -r 1 -m GET -l 3 [ 1692.591969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.154@tcp -r 1 -m GET -e local_error [ 1692.627610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.154@tcp -r 1 -m PUT -l 6 [ 1692.964608] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1694.394216] Lustre: DEBUG MARKER: Force oleg454-client.virtnet to discover 192.168.204.154@tcp (in background) [ 1694.518738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.154@tcp [ 1696.351322] Lustre: DEBUG MARKER: Force oleg454-server to discover 192.168.204.54@tcp [ 1697.631283] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1699.167817] Lustre: DEBUG MARKER: Wait for 72496 [ 1701.109476] Lustre: DEBUG MARKER: Finished wait on 72496 [ 1701.190726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1701.248322] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1701.275422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1701.628535] LNet: 72952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.633917] LNetError: 72952:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.701241] LNet: Removed LNI 192.168.204.54@tcp [ 1703.274939] Key type .llcrypt unregistered [ 1703.278379] Key type ._llcrypt unregistered [ 1721.210818] Key type ._llcrypt registered [ 1721.212814] Key type .llcrypt registered [ 1722.943891] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:14:34 (1767316474) [ 1723.787326] Key type .llcrypt unregistered [ 1723.789110] Key type ._llcrypt unregistered [ 1730.706268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1730.977788] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1730.982621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1730.989260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1733.605643] Key type ._llcrypt registered [ 1733.611272] Key type .llcrypt registered [ 1733.834722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.849805] alg: No test for adler32 (adler32-zlib) [ 1734.778983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.855287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.878635] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1735.631774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1735.648785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1744.907432] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:14:56 (1767316496) [ 1745.132631] LNet: 75021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.142372] LNetError: 75021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.148744] LNet: Removed LNI 192.168.204.54@tcp [ 1745.167270] LNet: Removed LNI 10.1.2.3@tcp [ 1746.017820] Key type .llcrypt unregistered [ 1746.024565] Key type ._llcrypt unregistered [ 1752.702451] Key type ._llcrypt registered [ 1752.705377] Key type .llcrypt registered [ 1752.847571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1752.864833] alg: No test for adler32 (adler32-zlib) [ 1753.819505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.871381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.893607] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1754.454996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1754.470161] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1755.286362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.54@tcp [ 1756.319336] LNetError: Unexpected error -22 connecting to 192.168.204.54@tcp at host 192.168.204.54:988 [ 1760.995512] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:15:12 (1767316512) [ 1761.078584] LNet: 76141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.087895] LNetError: 76141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.094422] LNet: Removed LNI 192.168.204.54@tcp [ 1761.115628] LNet: Removed LNI 10.1.2.3@tcp [ 1761.659911] Key type .llcrypt unregistered [ 1761.662377] Key type ._llcrypt unregistered [ 1768.157799] Key type ._llcrypt registered [ 1768.159710] Key type .llcrypt registered [ 1768.328309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1768.348987] alg: No test for adler32 (adler32-zlib) [ 1769.233692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.262549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1769.273814] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1770.010604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1770.036089] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 1770.641764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.54@tcp1 --nid 192.168.204.54@tcp2 [ 1774.041243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp1 192.168.204.54@tcp2 [ 1774.061153] LNetError: 76981:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.54@tcp1 to 192.168.204.54@tcp2 - Route table may be misconfigured [ 1774.088943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp2 192.168.204.54@tcp1 [ 1779.342841] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:15:31 (1767316531) [ 1779.493746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.537610] LNet: 77553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.551850] LNetError: 77553:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.575302] LNet: Removed LNI 192.168.204.54@tcp1 [ 1780.580706] LNetError: 77553:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.599788] LNet: Removed LNI 192.168.204.54@tcp2 [ 1780.650417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.701564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1780.717267] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1781.491239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1781.520741] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1782.404671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 1782.439694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.54@tcp [ 1782.468806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp1 192.168.204.54@tcp1 [ 1782.504663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp -d 192.168.204.54@tcp -e network_timeout [ 1782.529690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -e network_timeout [ 1782.559448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -e network_timeout [ 1782.596466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -e network_timeout [ 1782.628023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1782.640825] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1782.659504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1782.743526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1782.751104] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1782.849906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1787.863331] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:15:39 (1767316539) [ 1788.034406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.069552] LNet: 78415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.075564] LNetError: 78415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.087471] LNet: Removed LNI 192.168.204.54@tcp [ 1790.192480] LNet: Removed LNI 192.168.204.54@tcp1 [ 1790.232779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.314810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1790.492292] LNet: 78452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.801647] Key type .llcrypt unregistered [ 1790.803961] Key type ._llcrypt unregistered [ 1800.243547] Key type ._llcrypt registered [ 1800.248739] Key type .llcrypt registered [ 1801.326119] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:15:53 (1767316553) [ 1801.773555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.793624] alg: No test for adler32 (adler32-zlib) [ 1802.670275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.774316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1805.132237] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:15:57 (1767316557) [ 1805.336545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.357273] LNet: 79546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.375638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.418096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.433372] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1805.969305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1805.986804] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1806.760395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1806.780844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp1 [ 1806.808313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp1 [ 1812.112255] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:16:03 (1767316563) [ 1813.602237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg454-server [ 1815.121940] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:16:06 (1767316566) [ 1816.484608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg454-server [ 1817.748223] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:16:09 (1767316569) [ 1819.912293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg454-server [ 1821.355697] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:16:13 (1767316573) [ 1823.253826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg454-server [ 1825.094666] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:16:16 (1767316576) [ 1826.839452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg454-server [ 1828.380538] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:16:20 (1767316580) [ 1829.987921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg454-server [ 1831.688761] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:16:23 (1767316583) [ 1833.540361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg454-server [ 1835.273159] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:16:26 (1767316586) [ 1837.394462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg454-server [ 1839.066700] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:16:30 (1767316590) [ 1841.388669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg454-server [ 1842.920790] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:16:34 (1767316594) [ 1843.170624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.190669] LNet: 83424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.197524] LNetError: 83424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.207451] LNet: Removed LNI 192.168.204.54@tcp [ 1844.261901] LNet: Removed LNI 192.168.204.54@tcp1 [ 1844.297698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.376056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.395561] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1845.116491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1845.246223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1845.451180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.469871] LNet: 83571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.474827] LNetError: 83571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.481133] LNetError: 83571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1845.494984] LNet: Removed LNI 192.168.204.54@tcp [ 1845.518947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.585946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1845.609128] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1846.500457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1846.626650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.672626] LNetError: 83716:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.719272] LNet: Removed LNI 192.168.204.54@tcp [ 1847.781103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.868261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.913527] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1848.896279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1848.991633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1849.246927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.279815] LNet: 83864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.294123] LNet: 83864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1849.311752] LNetError: 83864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.335416] LNet: Removed LNI 192.168.204.54@tcp [ 1849.387933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.460595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1849.485934] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1850.329931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1851.760461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.852738] LNet: Removed LNI 192.168.204.54@tcp [ 1852.961683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.063678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1853.107387] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1854.127854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1854.251096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1854.560645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.593677] LNet: 84165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.599748] LNet: 84165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1854.607275] LNetError: 84165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.616478] LNetError: 84165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1855.669682] LNet: Removed LNI 192.168.204.54@tcp [ 1855.759399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.836342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1855.867994] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1856.772928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1857.005335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.085979] LNet: Removed LNI 192.168.204.54@tcp [ 1858.130764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.236652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.268034] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1858.929767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1858.987876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1859.212831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.265120] LNet: Removed LNI 192.168.204.54@tcp [ 1860.310541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.358317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1860.390278] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1861.143660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1861.370612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.449096] LNet: Removed LNI 192.168.204.54@tcp [ 1862.482127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.556770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1862.596634] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1863.472734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1863.580513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1864.932866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.950404] LNet: 84760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.960433] LNet: 84760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1864.973937] LNetError: 84760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.981805] LNetError: 84760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1866.024518] LNet: Removed LNI 192.168.204.54@tcp [ 1866.088967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.168452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1866.226173] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1866.951953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1867.247259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.326733] LNet: Removed LNI 192.168.204.54@tcp [ 1868.343415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.374878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.392631] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1869.023154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1869.128745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1869.411826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.502938] LNet: Removed LNI 192.168.204.54@tcp [ 1870.530759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.583823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1870.607668] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1871.212991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 1871.475738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.552510] LNet: Removed LNI 192.168.204.54@tcp [ 1872.586526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.671746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.710699] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1873.272951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1873.344672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.369213] LNet: Removed LNI 192.168.204.54@tcp [ 1873.389766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.443779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.464624] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1874.057708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1880.368415] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:17:12 (1767316632) [ 1880.622260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.656826] LNet: Removed LNI 192.168.204.54@tcp [ 1880.679194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.719269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1880.734915] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1881.638374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.672383] LNet: 86182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.675289] LNet: 86182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1881.682583] LNetError: 86182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.689913] LNetError: 86182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1881.703700] LNet: Removed LNI 192.168.204.54@tcp231 [ 1881.722184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.786735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1881.809051] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1881.811789] LNet: 86190:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1882.154869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1882.177971] LNet: Removed LNI 192.168.204.54@tcp231 [ 1882.190471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1882.207776] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1883.020718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.045336] LNet: Removed LNI 192.168.204.54@tcp231 [ 1883.064659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.133077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1883.160160] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/0] [ 1883.166640] LNet: 86343:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1883.597992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1883.634787] LNet: Removed LNI 192.168.204.54@tcp231 [ 1883.651255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1883.685744] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/0] [ 1884.647434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.682166] LNet: Removed LNI 192.168.204.54@tcp231 [ 1884.713331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.791668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1884.854413] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/60] [ 1884.871879] LNet: 86494:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1885.424424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1885.476728] LNet: Removed LNI 192.168.204.54@tcp231 [ 1885.509377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1885.536897] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/60] [ 1886.640847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.664300] LNet: Removed LNI 192.168.204.54@tcp231 [ 1886.686393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.778798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1886.807101] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1887.281124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1887.324440] LNet: Removed LNI 192.168.204.54@tcp231 [ 1887.343565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1887.373520] LNet: Added LNI 192.168.204.54@tcp231 [8/256/0/180] [ 1893.803926] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:17:25 (1767316645) [ 1893.919309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.955827] LNet: Removed LNI 192.168.204.54@tcp231 [ 1894.001313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.084687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1894.108413] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1894.816557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1894.844961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1894.870228] LNet: Removed LNI 192.168.204.54@tcp232 [ 1894.905654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1894.940068] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1895.754972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1895.806907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1895.849853] LNet: Removed LNI 192.168.204.54@tcp232 [ 1895.868361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1895.895905] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1896.656638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1896.691879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1896.715917] LNet: Removed LNI 192.168.204.54@tcp232 [ 1896.741216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1896.756682] LNet: Added LNI 192.168.204.54@tcp232 [8/256/0/180] [ 1897.386644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1897.411479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1897.435994] LNet: Removed LNI 192.168.204.54@tcp232 [ 1897.453857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1902.562460] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:17:34 (1767316654) [ 1903.546513] Key type .llcrypt unregistered [ 1903.549424] Key type ._llcrypt unregistered [ 1908.642755] Key type ._llcrypt registered [ 1908.644949] Key type .llcrypt registered [ 1908.793496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1908.802853] alg: No test for adler32 (adler32-zlib) [ 1909.716703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1909.750048] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1916.526507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 1923.952025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.983977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1924.002096] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1925.485852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_interrupt -r 2 [ 1925.537347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 2 [ 1925.572307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_interrupt -r 2 [ 1925.625145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 2 [ 1925.686103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_interrupt -r 2 [ 1925.811625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_interrupt -r 2 [ 1925.866421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_interrupt -r 2 [ 1925.935611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_interrupt -r 2 [ 1925.974192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1926.002268] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1926.005874] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1926.355316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1926.398837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1926.432975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1926.482858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1926.523420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1926.583871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1926.625824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1926.672314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1926.698156] LNet: 89142:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1926.735229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1926.797304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1926.861932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1926.906235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1926.959247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1927.043064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1927.702481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_dropped -r 2 [ 1927.746250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 2 [ 1927.783739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_dropped -r 2 [ 1927.824062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 2 [ 1927.849895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_dropped -r 2 [ 1927.890915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_dropped -r 2 [ 1927.937556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_dropped -r 2 [ 1927.976173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_dropped -r 2 [ 1928.018772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1928.036292] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1928.381986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1928.447146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1928.488215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1928.533353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1928.573867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1928.613062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1928.657165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1928.703190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1928.721606] LNet: 89262:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1928.754962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1928.796717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1928.821421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1928.848292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1928.869900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1928.900983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1929.423938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_aborted -r 2 [ 1929.459439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 2 [ 1929.504564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_aborted -r 2 [ 1929.544282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 2 [ 1929.585712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_aborted -r 2 [ 1929.619574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_aborted -r 2 [ 1929.659820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_aborted -r 2 [ 1929.716253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_aborted -r 2 [ 1929.779399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1929.794354] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1930.084501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1930.122518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1930.160290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1930.193884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1930.242367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1930.280474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1930.334535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1930.383697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1930.420604] LNet: 89383:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1930.466463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1930.515359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1930.551343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1930.595305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1930.643166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1930.702849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1931.285496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_no_route -r 2 [ 1931.315960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 2 [ 1931.351290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_no_route -r 2 [ 1931.378552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 2 [ 1931.417349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_no_route -r 2 [ 1931.441809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_no_route -r 2 [ 1931.464549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_no_route -r 2 [ 1931.496831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_no_route -r 2 [ 1931.536703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1931.762917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1931.786659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1931.818874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1931.842130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1931.867943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1931.892245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1931.920396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1931.942600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1931.981849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1932.012275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1932.037199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1932.073306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1932.095086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1932.117229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1932.643804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e local_timeout -r 2 [ 1932.689495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 2 [ 1932.742922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e local_timeout -r 2 [ 1932.804678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 2 [ 1932.840706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e local_timeout -r 2 [ 1932.881386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e local_timeout -r 2 [ 1932.926795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e local_timeout -r 2 [ 1932.981849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e local_timeout -r 2 [ 1933.054242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1933.082044] LNet: There was an unexpected network error while writing to 192.168.204.154: rc = -22 [ 1933.094660] LNet: Skipped 1 previous similar message [ 1933.379654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1933.415641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1933.444877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1933.476697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1933.516526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1933.556735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1933.588917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1933.610422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1933.628736] LNet: 89623:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1933.634654] LNet: 89623:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1933.665271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1933.701568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1933.731365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1933.748400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1933.784933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1933.827629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1934.470591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp -m GET -e remote_dropped -r 2 [ 1934.518786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 2 [ 1934.589249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp -m GET -e remote_dropped -r 2 [ 1934.645770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 2 [ 1934.710287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp -m GET -e remote_dropped -r 2 [ 1934.772980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.154@tcp1 -m GET -e remote_dropped -r 2 [ 1934.818498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp -m GET -e remote_dropped -r 2 [ 1934.898605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.54@tcp1 -d 192.168.204.54@tcp1 -m GET -e remote_dropped -r 2 [ 1934.937498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.54@tcp 192.168.204.154@tcp [ 1934.966788] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 1935.322971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1935.360221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1935.403313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1935.444316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1935.485875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1935.530724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1935.568584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1935.608527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1935.684583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1935.733853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1935.775932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1935.808143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1935.832916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1935.875881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1942.813152] LNet: 89817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.817451] LNetError: 89817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.820754] LNet: Removed LNI 192.168.204.54@tcp [ 1942.835325] LNet: Removed LNI 192.168.204.54@tcp1 [ 1943.142751] Key type .llcrypt unregistered [ 1943.144140] Key type ._llcrypt unregistered [ 1958.097847] Key type ._llcrypt registered [ 1958.099499] Key type .llcrypt registered [ 1959.282638] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:18:30 (1767316710) [ 1959.661722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.685143] alg: No test for adler32 (adler32-zlib) [ 1960.591565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.648945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.669989] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1961.382670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1961.408917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.54@tcp [ 1961.437587] LNetError: 90791:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1961.497572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.54@tcp [ 1961.596538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1966.887834] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:18:38 (1767316718) [ 1967.174305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.191756] LNet: 91380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.195282] LNetError: 91380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.211956] LNet: Removed LNI 192.168.204.54@tcp [ 1967.226548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.284485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1967.306925] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1967.943075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1967.958317] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 1968.669177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.54@tcp [ 1968.684533] LNetError: 91646:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.54@tcp to recovery [ 1969.311369] LNet: 1 local NIs in recovery (showing 1): 192.168.204.54@tcp [ 1969.317153] LNet: There was an unexpected network error while writing to 192.168.204.54: rc = -22 [ 1969.946239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.54@tcp [ 1978.274750] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:18:49 (1767316729) [ 1979.997033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg454-server [ 1981.627667] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:18:53 (1767316733) [ 1981.765920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.813249] LNet: 92595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.821653] LNetError: 92595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.836851] LNet: Removed LNI 192.168.204.54@tcp [ 1981.871323] LNet: Removed LNI 192.168.204.54@tcp1 [ 1981.917563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.996900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.047799] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1982.884710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1982.963551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1983.012194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1983.132263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1983.196392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1983.231162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1989.405696] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:19:01 (1767316741) [ 1989.542744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.611628] LNet: 93323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.624930] LNetError: 93323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.632161] LNetError: 93323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1989.671032] LNet: Removed LNI 192.168.204.54@tcp [ 1989.703794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.751225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.767797] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 1996.080900] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:19:07 (1767316747) [ 1996.199501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.254676] LNet: 94029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.262394] LNetError: 94029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.317248] LNet: Removed LNI 192.168.204.54@tcp [ 1996.351295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.418857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.436582] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2002.864799] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:19:14 (1767316754) [ 2004.294526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2006.079538] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:19:17 (1767316757) [ 2006.482399] LNet: 95067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.490773] LNetError: 95067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.506167] LNet: Removed LNI 192.168.204.54@tcp [ 2007.186692] Key type .llcrypt unregistered [ 2007.188837] Key type ._llcrypt unregistered [ 2012.607404] Key type ._llcrypt registered [ 2012.609575] Key type .llcrypt registered [ 2012.800585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2012.813945] alg: No test for adler32 (adler32-zlib) [ 2013.751136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2013.816927] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2021.512372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2026.308890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.154@tcp [ 2026.681056] LNet: 95479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.696933] LNetError: 95479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.725268] LNet: Removed LNI 192.168.204.54@tcp [ 2027.276818] Key type .llcrypt unregistered [ 2027.278799] Key type ._llcrypt unregistered [ 2038.512522] Key type ._llcrypt registered [ 2038.515830] Key type .llcrypt registered [ 2039.825903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2041.937150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2043.757349] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:19:55 (1767316795) [ 2044.264809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.314207] alg: No test for adler32 (adler32-zlib) [ 2045.308925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.380837] LNet: 96562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.833648] Key type .llcrypt unregistered [ 2045.837156] Key type ._llcrypt unregistered [ 2051.176875] Key type ._llcrypt registered [ 2051.178343] Key type .llcrypt registered [ 2051.566886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.618248] alg: No test for adler32 (adler32-zlib) [ 2052.533194] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2052.536761] LNet: 96814:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2052.545284] LNet: 96814:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2052.598683] LNetError: 96826:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.55@tcp is being used as a gateway but routing feature is not turned on [ 2052.622317] LNetError: 96826:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.55@tcp has gone from up to down [ 2053.418164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.488896] LNet: 96872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.508678] LNetError: 96872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.568150] LNet: Removed LNI 192.168.204.54@tcp [ 2053.610499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.779497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2053.818427] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2053.832837] LNet: 96879:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2053.843162] LNet: 96879:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 2053.872425] LNetError: 96876:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.56@tcp is being used as a gateway but routing feature is not turned on [ 2053.885292] LNetError: 96876:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2053.909420] LNetError: 96876:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.56@tcp has gone from up to down [ 2053.917911] LNetError: 96876:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2053.951897] LNetError: 96879:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.56@tcp is a gateway. Can not delete it [ 2053.960201] LNetError: 96879:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.204.56@tcp is a router. Can not be deleted [ 2053.974141] LNet: 96879:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2061.817629] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:20:13 (1767316813) [ 2063.963629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg454-server [ 2065.812727] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:20:17 (1767316817) [ 2067.632978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg454-server [ 2069.458089] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:20:20 (1767316820) [ 2071.050331] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 2072.847190] Key type lgssc registered [ 2074.112425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.250064] LNetError: 96876:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 2113.268055] LNetError: 96876:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 2134.821919] Key type lgssc unregistered [ 2135.183642] LNet: 99769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.196904] LNetError: 99769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.229702] LNet: Removed LNI 192.168.204.54@tcp [ 2136.874617] Key type .llcrypt unregistered [ 2136.876947] Key type ._llcrypt unregistered [ 2151.975815] Key type ._llcrypt registered [ 2151.977833] Key type .llcrypt registered [ 2153.363416] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:21:44 (1767316904) [ 2153.759624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2153.787095] alg: No test for adler32 (adler32-zlib) [ 2154.735231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.797516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2154.848188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2154.911718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2154.960871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2160.374130] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:21:52 (1767316912) [ 2160.589099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.643687] LNet: 101173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.667635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.806103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.849683] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2161.803348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.819622] LNet: 101317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.825635] LNetError: 101317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.842345] LNet: Removed LNI 192.168.204.54@tcp [ 2161.864520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.937500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2161.966713] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2162.400336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.422785] LNetError: 101345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.439971] LNet: Removed LNI 192.168.204.54@tcp [ 2162.460275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.555865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2162.599238] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2169.793834] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:22:01 (1767316921) [ 2170.028192] LNet: 101936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.032429] LNet: 101936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2170.037835] LNetError: 101936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.051237] LNet: Removed LNI 192.168.204.54@tcp [ 2170.868668] Key type .llcrypt unregistered [ 2170.870081] Key type ._llcrypt unregistered [ 2179.120982] Key type ._llcrypt registered [ 2179.126301] Key type .llcrypt registered [ 2179.435235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2179.466775] alg: No test for adler32 (adler32-zlib) [ 2180.500944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.629384] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2180.635321] LNet: *** cfs_fail_loc=e003, val=100*** [ 2181.469783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2181.493912] LNet: *** cfs_fail_loc=e003, val=100*** [ 2181.495635] LNet: Skipped 2 previous similar messages [ 2181.504252] Ping reply received with mapping: [ 2181.507462] found metadata reply NI[192.168.204.54@tcp] [ 2181.512166] 36 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2181.515503] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2181.518937] 5a 5a 5a 5a 5a 5a 5a 5a [ 2181.642669] LNet: 102330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.652845] LNetError: 102330:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.664967] LNet: Removed LNI 192.168.204.54@tcp [ 2182.425303] Key type .llcrypt unregistered [ 2182.427725] Key type ._llcrypt unregistered [ 2192.802962] Key type ._llcrypt registered [ 2192.805231] Key type .llcrypt registered [ 2193.763216] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:22:25 (1767316945) [ 2194.307700] Key type .llcrypt unregistered [ 2194.310226] Key type ._llcrypt unregistered [ 2200.028272] Key type ._llcrypt registered [ 2200.032712] Key type .llcrypt registered [ 2200.316885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2200.341705] alg: No test for adler32 (adler32-zlib) [ 2201.365764] LNetError: 103367:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2201.817492] Key type .llcrypt unregistered [ 2201.820326] Key type ._llcrypt unregistered [ 2206.096423] Key type ._llcrypt registered [ 2206.102145] Key type .llcrypt registered [ 2207.126228] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:22:38 (1767316958) [ 2207.773224] Key type .llcrypt unregistered [ 2207.774870] Key type ._llcrypt unregistered [ 2213.174312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2213.185049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2215.774745] Key type ._llcrypt registered [ 2215.776978] Key type .llcrypt registered [ 2215.955587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2215.965663] alg: No test for adler32 (adler32-zlib) [ 2216.927626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.026314] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2226.299585] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:22:58 (1767316978) [ 2226.516983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.565537] LNet: 105137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.580296] LNetError: 105137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2226.610211] LNet: Removed LNI 10.1.2.3@tcp [ 2226.631693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.342956] Key type .llcrypt unregistered [ 2227.344823] Key type ._llcrypt unregistered [ 2232.920392] Key type ._llcrypt registered [ 2232.922731] Key type .llcrypt registered [ 2233.062405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2233.072727] alg: No test for adler32 (adler32-zlib) [ 2233.961664] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2234.013918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2234.114293] LNet: 105424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.118756] LNetError: 105424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.134699] LNet: Removed LNI 192.168.204.54@tcp [ 2234.736844] Key type .llcrypt unregistered [ 2234.739689] Key type ._llcrypt unregistered [ 2244.841364] Key type ._llcrypt registered [ 2244.843931] Key type .llcrypt registered [ 2246.211975] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:23:17 (1767316997) [ 2246.974553] Key type .llcrypt unregistered [ 2246.977328] Key type ._llcrypt unregistered [ 2252.963816] Key type ._llcrypt registered [ 2252.966129] Key type .llcrypt registered [ 2253.131341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2253.144121] alg: No test for adler32 (adler32-zlib) [ 2258.926902] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:23:30 (1767317010) [ 2259.155502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.186078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.238681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2259.278509] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2259.952962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2259.983190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2260.012737] LNet: Removed LNI 192.168.204.54@tcp [ 2260.229287] LNet: 107160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.485766] Key type .llcrypt unregistered [ 2260.488118] Key type ._llcrypt unregistered [ 2270.107921] Key type ._llcrypt registered [ 2270.110169] Key type .llcrypt registered [ 2271.302980] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:23:42 (1767317022) [ 2271.683870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.699938] alg: No test for adler32 (adler32-zlib) [ 2272.601058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.650272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.672238] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2273.419135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2273.451960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.54@tcp [ 2273.468909] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2273.472995] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2273.476244] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2273, 0, 0 [ 2273.481204] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2273.483900] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2273.489832] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2273.493396] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2273, 0, 0 [ 2273.500307] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2273.504727] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2273.508818] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2273.511988] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2273, 0, 2323 [ 2273.518644] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2273.520975] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2273.524496] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2273.528503] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2273, 0, 2323 [ 2273.533688] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2279.389740] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:23:51 (1767317031) [ 2279.894860] LNet: 108696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.905908] LNetError: 108696:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.920900] LNet: Removed LNI 192.168.204.54@tcp [ 2280.548734] Key type .llcrypt unregistered [ 2280.555045] Key type ._llcrypt unregistered [ 2285.784934] Key type ._llcrypt registered [ 2285.790838] Key type .llcrypt registered [ 2286.007836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2286.018190] alg: No test for adler32 (adler32-zlib) [ 2286.956543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2286.986648] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2293.496580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2301.535953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.563668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2301.581763] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 2302.707405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2302.894688] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2302.897370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2302.902278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2305.200369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2305.218350] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2314.193860] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:24:26 (1767317066) [ 2314.486807] LNet: 110195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.497269] LNetError: 110195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.509691] LNet: Removed LNI 192.168.204.54@tcp [ 2315.551968] LNet: Removed LNI 192.168.204.54@tcp1 [ 2315.559478] LNetError: 110195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2315.572076] LNetError: 110195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2315.589984] LNet: Removed LNI 10.1.2.3@tcp99 [ 2316.184797] Key type .llcrypt unregistered [ 2316.188502] Key type ._llcrypt unregistered [ 2322.897710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2322.913750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2325.529952] Key type ._llcrypt registered [ 2325.535157] Key type .llcrypt registered [ 2325.764778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2325.788788] alg: No test for adler32 (adler32-zlib) [ 2326.810381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.889200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.925243] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2327.683166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2327.712031] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2329.614778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2329.657471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2329.728975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2329.746522] LNetError: 110936:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2329.771654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2329.806712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2330.127096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2330.199988] LNet: 110956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.204817] LNetError: 110956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.209719] LNet: Removed LNI 192.168.204.54@tcp [ 2330.219492] LNet: Removed LNI 10.1.2.3@tcp [ 2330.780550] Key type .llcrypt unregistered [ 2330.782382] Key type ._llcrypt unregistered [ 2340.860079] Key type ._llcrypt registered [ 2340.861474] Key type .llcrypt registered [ 2341.909308] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:24:53 (1767317093) [ 2342.259720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.266876] alg: No test for adler32 (adler32-zlib) [ 2343.281487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.350510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.372098] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2343.903642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.54@tcp [ 2343.971377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg454-client@tcp [ 2349.476969] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:25:01 (1767317101) [ 2349.665586] LNet: 112485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.678963] LNetError: 112485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.713869] LNet: Removed LNI 192.168.204.54@tcp [ 2350.309199] Key type .llcrypt unregistered [ 2350.313519] Key type ._llcrypt unregistered [ 2356.179974] Key type ._llcrypt registered [ 2356.184050] Key type .llcrypt registered [ 2356.351700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.366312] alg: No test for adler32 (adler32-zlib) [ 2357.261146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2357.287472] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2357.320726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2362.706646] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:25:14 (1767317114) [ 2364.025677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2365.987852] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:25:17 (1767317117) [ 2367.320842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2368.870757] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:25:20 (1767317120) [ 2368.970462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.988543] LNet: 113894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.993875] LNetError: 113894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.011370] LNet: Removed LNI 192.168.204.54@tcp [ 2369.030288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.099446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.123826] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2369.878991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2369.930505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2376.387637] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:25:28 (1767317128) [ 2376.721525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.738989] LNet: 114611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.744828] LNetError: 114611:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.771535] LNet: Removed LNI 192.168.204.54@tcp [ 2377.791731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.868362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2377.903838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2378.238695] LNet: 114646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.652332] Key type .llcrypt unregistered [ 2378.657613] Key type ._llcrypt unregistered [ 2390.260728] Key type ._llcrypt registered [ 2390.262811] Key type .llcrypt registered [ 2391.306555] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:25:43 (1767317143) [ 2391.695613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2391.721956] alg: No test for adler32 (adler32-zlib) [ 2392.653151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.714433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.747753] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2393.384821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2393.447088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2399.871287] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:25:51 (1767317151) [ 2400.085515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.126947] LNet: 116162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.138981] LNetError: 116162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.170586] LNet: Removed LNI 192.168.204.54@tcp [ 2400.203508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.248337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2400.269365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2405.411516] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:25:57 (1767317157) [ 2405.591729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.619152] LNet: 116743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.635553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.671893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2405.981554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2406.014800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2406.053291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2406.368597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2406.400310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2406.494680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.559149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.697591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2407.156754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2407.186407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2407.276550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2407.743371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2407.789963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2414.794822] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:26:06 (1767317166) [ 2415.217351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.278701] LNet: 117377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.296607] LNet: 117377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2415.384920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.502228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.574802] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2416.533770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2416.571704] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2417.555879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2417.636978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2417.820685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.845668] LNet: 117656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.849671] LNetError: 117656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.858918] LNet: Removed LNI 192.168.204.54@tcp [ 2417.867546] LNet: Removed LNI 192.168.204.54@tcp2 [ 2417.889740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.947126] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2417.952752] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2417.957496] LNet: 117662:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.546460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2418.787763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2419.807278] LNet: Removed LNI 192.168.204.54@tcp [ 2419.808610] LNetError: 117699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.818827] LNetError: 117699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2419.832246] LNet: Removed LNI 192.168.204.54@tcp2 [ 2419.861773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.910479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2419.927273] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2420.475124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2420.499083] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2421.660275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2421.709769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2421.765392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2421.925452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.945627] LNet: 117984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.951313] LNet: 117984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2421.957282] LNetError: 117984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.962470] LNet: Removed LNI 192.168.204.54@tcp [ 2423.017903] LNet: Removed LNI 192.168.204.54@tcp2 [ 2423.096755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.171357] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2423.176517] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2423.179831] LNet: 117990:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.505818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2423.766349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.781386] LNet: Removed LNI 192.168.204.54@tcp [ 2424.806467] LNet: Removed LNI 192.168.204.54@tcp2 [ 2424.823325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.935231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.978649] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2425.720195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2425.743133] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2426.731213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2426.793950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2427.149388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2427.184300] LNetError: 118306:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.187195] LNetError: 118306:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2427.202342] LNet: Removed LNI 192.168.204.54@tcp [ 2427.215700] LNet: Removed LNI 192.168.204.54@tcp2 [ 2427.242808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.314577] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2427.321076] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2427.328215] LNet: 118312:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.794536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2427.998060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.023549] LNet: Removed LNI 192.168.204.54@tcp [ 2429.037168] LNet: Removed LNI 192.168.204.54@tcp2 [ 2429.053579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.113053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.159668] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2429.883901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2429.902348] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2430.802566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2430.849796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2430.908429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2431.123928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.139891] LNet: 118634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.142606] LNet: 118634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2432.159581] LNet: Removed LNI 192.168.204.54@tcp [ 2433.191292] LNet: Removed LNI 192.168.204.54@tcp2 [ 2433.243938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.380350] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2433.394951] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2433.408934] LNet: 118640:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.790148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2433.902284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.913830] LNet: Removed LNI 192.168.204.54@tcp [ 2434.982125] LNet: Removed LNI 192.168.204.54@tcp2 [ 2435.010195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.062603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.096690] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2435.772578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2435.798505] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2436.542644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2436.566842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2436.622969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2436.942868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.964618] LNetError: 118974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.969233] LNetError: 118974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2436.974666] LNet: Removed LNI 192.168.204.54@tcp [ 2436.983719] LNet: Removed LNI 192.168.204.54@tcp2 [ 2437.005981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.088619] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2437.105860] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2437.673862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2437.983636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2439.007240] LNet: Removed LNI 192.168.204.54@tcp [ 2439.022272] LNet: Removed LNI 192.168.204.54@tcp2 [ 2439.093636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2439.139498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2439.155888] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2439.700696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2439.718557] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2440.400802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2440.426459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2440.457922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2440.508617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2440.786708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.823165] LNet: Removed LNI 192.168.204.54@tcp [ 2442.855907] LNet: Removed LNI 192.168.204.54@tcp2 [ 2442.885866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.978093] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2442.982627] LNet: Added LNI 192.168.204.54@tcp2 [8/256/0/180] [ 2442.986503] LNet: 119340:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2442.990454] LNet: 119340:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 2443.686258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2449.703628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2451.301509] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:26:42 (1767317202) [ 2451.460333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.481532] LNet: 120097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.484637] LNet: 120097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2451.493144] LNet: Removed LNI 192.168.204.54@tcp [ 2452.523504] LNet: Removed LNI 192.168.204.54@tcp2 [ 2452.560077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2452.830626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2452.861174] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2453.493992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2453.514598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2454.199725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.221350] LNetError: 120372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.225946] LNetError: 120372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2454.230661] LNet: Removed LNI 192.168.204.54@tcp [ 2454.239930] LNet: Removed LNI 10.1.2.3@tcp [ 2454.254477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.311927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2454.328415] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2454.332865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2454.779478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2454.798623] LNet: Removed LNI 192.168.204.54@tcp [ 2454.807142] LNet: Removed LNI 10.1.2.3@tcp [ 2460.171158] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:26:51 (1767317211) [ 2460.259662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.330053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.443143] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2460.552852] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2460.562811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2460.569966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2462.925224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2462.963181] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2463.643526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2463.667869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2465.522762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.54@tcp [ 2465.587463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2465.656791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.54@tcp -m PUT -e local_timeout -r 1 [ 2465.724045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2465.793275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2465.794900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2466.045237] LNetError: 121038:0:(lib-move.c:4947:lnet_parse()) 192.168.204.54@tcp, src 192.168.204.54@tcp: Dropping PUT (error -108 looking up sender) [ 2468.127953] LNet: Removed LNI 192.168.204.54@tcp [ 2468.138903] LNet: Removed LNI 10.1.2.3@tcp [ 2468.590985] Key type .llcrypt unregistered [ 2468.593802] Key type ._llcrypt unregistered [ 2477.904464] Key type ._llcrypt registered [ 2477.907476] Key type .llcrypt registered [ 2478.899576] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:27:10 (1767317230) [ 2479.301752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.315326] alg: No test for adler32 (adler32-zlib) [ 2480.232873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.522514] LNet: 122345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.049490] Key type .llcrypt unregistered [ 2481.052527] Key type ._llcrypt unregistered [ 2486.379206] Key type ._llcrypt registered [ 2486.381343] Key type .llcrypt registered [ 2486.531856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2486.541874] alg: No test for adler32 (adler32-zlib) [ 2487.464928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2487.485717] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2494.767602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2494.963600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2495.564546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.154@tcp [ 2495.714369] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 2495.764881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2495.816043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2495.849708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2495.879151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2495.907338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2495.944487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2495.974594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2495.996481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2496.015403] LNet: 122802:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2496.033344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2496.059687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2496.091466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2496.119866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2496.160239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2496.199043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2496.712596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2496.980321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2496.999510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2497.021283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2497.042589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2497.060127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2497.111256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2497.172394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2497.219907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2497.245684] LNet: 122919:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2497.274800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2497.302992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2497.330621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2497.385160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2497.447316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2497.483942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.068701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.154@tcp [ 2498.233795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2498.258116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2498.297801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2498.349698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.383885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.407600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.428611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.450707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.465625] LNet: 123036:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.480369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.506383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.528712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.554826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.584716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.610844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2499.168249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2499.392311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2499.412920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2499.433830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2499.478331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2499.520854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2499.560213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2499.585864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2499.607350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2499.641515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2499.683996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2499.732336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2499.769387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2499.814378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2499.846522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2500.109735] LNet: 123199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.114687] LNetError: 123199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2500.128876] LNet: Removed LNI 192.168.204.54@tcp [ 2500.477838] Key type .llcrypt unregistered [ 2500.481982] Key type ._llcrypt unregistered [ 2512.984609] Key type ._llcrypt registered [ 2512.986623] Key type .llcrypt registered [ 2513.912258] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:27:45 (1767317265) [ 2514.387692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.412874] alg: No test for adler32 (adler32-zlib) [ 2515.290789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2515.565315] LNet: 124082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.091357] Key type .llcrypt unregistered [ 2516.092948] Key type ._llcrypt unregistered [ 2521.561119] Key type ._llcrypt registered [ 2521.564219] Key type .llcrypt registered [ 2521.816902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2521.829482] alg: No test for adler32 (adler32-zlib) [ 2522.756665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2522.782237] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2530.181311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.154@tcp [ 2538.283311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.319767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2538.343544] LNet: Added LNI 192.168.204.54@tcp1 [8/256/0/180] [ 2539.060791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2539.590375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.154@tcp [ 2539.856924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2539.893923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2539.935939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2539.973946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2540.005706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2540.036649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2540.073889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2540.109632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2540.137954] LNet: 124729:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2540.163435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2540.198361] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.154@tcp [ 2540.211965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2540.241958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2540.276981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2540.300318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2540.325375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2540.928423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2541.191045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2541.250239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2541.298619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2541.345128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2541.403416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2541.442422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2541.488837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2541.536657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2541.573405] LNet: 124847:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2541.604554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2541.657966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2541.707911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2541.747834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2541.792311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2541.847346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.490094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.154@tcp [ 2542.723910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2542.744599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2542.770593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2542.795638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2542.826414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2542.854462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2542.900451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2542.942986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2542.966127] LNet: 124964:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2542.984743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.012625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.052946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.088949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.108487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.136623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2543.722676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2543.895255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2543.913106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2543.930371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2543.948224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.965981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.983060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.008215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.029695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.048185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.078782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.110858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.154869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.205325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.237180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2548.516745] LNet: 125160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.522572] LNetError: 125160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.531277] LNet: Removed LNI 192.168.204.54@tcp [ 2548.541049] LNet: Removed LNI 192.168.204.54@tcp1 [ 2548.964644] Key type .llcrypt unregistered [ 2548.966660] Key type ._llcrypt unregistered [ 2565.326969] Key type ._llcrypt registered [ 2565.329303] Key type .llcrypt registered [ 2566.752300] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:28:38 (1767317318) [ 2574.217483] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:28:45 (1767317325) [ 2582.322947] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:28:53 (1767317333) [ 2589.088242] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:29:00 (1767317340) [ 2596.974385] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:29:08 (1767317348) [ 2603.351367] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:29:15 (1767317355) [ 2608.975394] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:29:20 (1767317360) [ 2615.395723] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:29:27 (1767317367) [ 2621.731608] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:29:33 (1767317373) [ 2628.754574] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:29:40 (1767317380) [ 2634.915664] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:29:46 (1767317386) [ 2641.870986] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:29:53 (1767317393) [ 2647.775442] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:29:59 (1767317399) [ 2654.846393] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:30:06 (1767317406) [ 2661.135250] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:30:13 (1767317413) [ 2666.683143] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:30:18 (1767317418) [ 2673.653829] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:30:25 (1767317425) [ 2679.828404] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:30:31 (1767317431) [ 2680.135698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2680.150028] alg: No test for adler32 (adler32-zlib) [ 2681.028133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2681.102492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2681.115891] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2681.880896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2681.904240] LNet: 135866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.910946] LNetError: 135866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.930341] LNet: Removed LNI 192.168.204.54@tcp [ 2681.950147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2682.011141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2682.039849] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2687.945739] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:30:39 (1767317439) [ 2694.711990] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:30:46 (1767317446) [ 2702.054607] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:30:53 (1767317453) [ 2709.639289] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:31:01 (1767317461) [ 2716.796806] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:31:08 (1767317468) [ 2723.377425] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:31:14 (1767317474) [ 2730.059903] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:31:21 (1767317481) [ 2737.151464] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:31:28 (1767317488) [ 2743.850499] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:31:35 (1767317495) [ 2750.856862] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:31:42 (1767317502) [ 2757.232466] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:31:48 (1767317508) [ 2763.325897] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:31:55 (1767317515) [ 2769.245131] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2406 sec ========== 20:32:00 (1767317520) [ 2769.497606] LNet: 143332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2769.501917] LNetError: 143332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2769.512861] LNet: Removed LNI 192.168.204.54@tcp [ 2769.931386] Key type .llcrypt unregistered [ 2769.932913] Key type ._llcrypt unregistered [ 2782.397963] Key type ._llcrypt registered [ 2782.399747] Key type .llcrypt registered [ 2782.776939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2782.784817] alg: No test for adler32 (adler32-zlib) [ 2783.832208] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 2784.055540] LNet: Added LNI 192.168.204.54@tcp [8/256/0/180] [ 2785.719162] Key type lgssc registered [ 2787.175080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.680454] Lustre: Mounted lustre-client [ 2880.838709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2892.988404] LustreError: 146651:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a4385367000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2893.011405] LustreError: 146651:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2893.055611] Lustre: Unmounted lustre-client [ 2924.034361] Key type lgssc unregistered [ 2924.212184] LNet: 147243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2924.216657] LNetError: 147243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.228329] LNet: Removed LNI 192.168.204.54@tcp [ 2924.843185] Key type .llcrypt unregistered [ 2924.852024] Key type ._llcrypt unregistered