[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 581253670 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002384] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007759] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010150] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.013032] *** VALIDATE selinux *** [ 0.021501] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025663] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] *** VALIDATE tmpfs *** [ 0.030199] *** VALIDATE proc *** [ 0.031273] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.034145] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040947] debug: unmapping init [mem 0xffffffffae859000-0xffffffffae860fff] [ 0.042912] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049020] ... fixed-purpose events: 3 [ 0.050018] ... event mask: 000000070000000f [ 0.051312] rcu: Hierarchical SRCU implementation. [ 0.053432] smp: Bringing up secondary CPUs ... [ 0.054444] x86: Booting SMP configuration: [ 0.055031] .... node #0, CPUs: #1 #2 #3 [ 0.058669] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061025] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.133702] node 0 deferred pages initialised in 69ms [ 0.137009] devtmpfs: initialized [ 0.138229] x86/mm: Memory block size: 128MB [ 0.140741] gcov: version magic: 0x41383552 [ 0.142104] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144336] pinctrl core: initialized pinctrl subsystem [ 0.145193] [ 0.145758] ************************************************************* [ 0.146012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147010] ** ** [ 0.148014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149016] ** ** [ 0.150014] ** This means that this kernel is built to expose internal ** [ 0.151015] ** IOMMU data structures, which may compromise security on ** [ 0.152016] ** your system. ** [ 0.153016] ** ** [ 0.154015] ** If you see this message and you are not debugging the ** [ 0.155014] ** kernel, report this immediately to your vendor! ** [ 0.156018] ** ** [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158015] ************************************************************* [ 0.159699] NET: Registered protocol family 16 [ 0.160457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163453] cpuidle: using governor menu [ 0.166092] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168597] PCI: Using configuration type 1 for base access [ 0.171135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182168] cryptd: max_cpu_qlen set to 1000 [ 0.184266] ACPI: Added _OSI(Module Device) [ 0.185010] ACPI: Added _OSI(Processor Device) [ 0.186015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190549] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193485] ACPI: Interpreter enabled [ 0.194055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195011] ACPI: Using IOAPIC for interrupt routing [ 0.196117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197363] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205409] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210048] acpiphp: Slot [2] registered [ 0.211128] acpiphp: Slot [5] registered [ 0.212108] acpiphp: Slot [6] registered [ 0.213118] acpiphp: Slot [3] registered [ 0.214113] acpiphp: Slot [4] registered [ 0.215109] acpiphp: Slot [7] registered [ 0.216170] acpiphp: Slot [8] registered [ 0.217096] acpiphp: Slot [9] registered [ 0.218140] acpiphp: Slot [10] registered [ 0.219103] acpiphp: Slot [11] registered [ 0.220102] acpiphp: Slot [12] registered [ 0.221134] acpiphp: Slot [13] registered [ 0.222117] acpiphp: Slot [14] registered [ 0.223122] acpiphp: Slot [15] registered [ 0.224123] acpiphp: Slot [16] registered [ 0.225104] acpiphp: Slot [17] registered [ 0.226092] acpiphp: Slot [18] registered [ 0.227107] acpiphp: Slot [19] registered [ 0.228123] acpiphp: Slot [20] registered [ 0.229111] acpiphp: Slot [21] registered [ 0.230149] acpiphp: Slot [22] registered [ 0.231098] acpiphp: Slot [23] registered [ 0.232076] acpiphp: Slot [24] registered [ 0.233098] acpiphp: Slot [25] registered [ 0.234103] acpiphp: Slot [26] registered [ 0.235178] acpiphp: Slot [27] registered [ 0.236103] acpiphp: Slot [28] registered [ 0.237133] acpiphp: Slot [29] registered [ 0.238093] acpiphp: Slot [30] registered [ 0.239105] acpiphp: Slot [31] registered [ 0.240059] PCI host bridge to bus 0000:00 [ 0.241016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.242020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.245033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.248034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.251028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.254823] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258385] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266024] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.270056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.273031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279653] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.283134] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.290000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.304658] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.309015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.315414] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.322017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.328017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.345027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.358113] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.365000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.373018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.402033] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417844] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.421398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.423404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.426389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.429258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.433104] iommu: Default domain type: Passthrough [ 0.436545] SCSI subsystem initialized [ 0.438222] ACPI: bus type USB registered [ 0.441209] usbcore: registered new interface driver usbfs [ 0.444137] usbcore: registered new interface driver hub [ 0.446136] usbcore: registered new device driver usb [ 0.448332] pps_core: LinuxPPS API ver. 1 registered [ 0.450027] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.454095] PTP clock support registered [ 0.457123] EDAC MC: Ver: 3.0.0 [ 0.459179] PCI: Using ACPI for IRQ routing [ 0.461946] NetLabel: Initializing [ 0.463020] NetLabel: domain hash size = 128 [ 0.465021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.467120] NetLabel: unlabeled traffic allowed by default [ 0.470189] vgaarb: loaded [ 0.472350] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.474027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.482000] clocksource: Switched to clocksource kvm-clock [ 0.598520] VFS: Disk quotas dquot_6.6.0 [ 0.600714] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.603344] *** VALIDATE ramfs *** [ 0.604185] *** VALIDATE hugetlbfs *** [ 0.605911] pnp: PnP ACPI init [ 0.608029] pnp: PnP ACPI: found 6 devices [ 0.623415] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.628070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.630172] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.632280] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.634763] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.637300] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.639992] NET: Registered protocol family 2 [ 0.642194] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.647665] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.651362] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.656709] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.660157] TCP: Hash tables configured (established 65536 bind 65536) [ 0.663389] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.666907] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.670399] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676735] NET: Registered protocol family 1 [ 0.682336] RPC: Registered named UNIX socket transport module. [ 0.684898] RPC: Registered udp transport module. [ 0.686759] RPC: Registered tcp transport module. [ 0.688672] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.691369] NET: Registered protocol family 44 [ 0.693199] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.695603] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.700017] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.702464] PCI: CLS 0 bytes, default 64 [ 0.705151] Unpacking initramfs... [ 2.121534] debug: unmapping init [mem 0xffff8d7d7cc64000-0xffff8d7d7ffcffff] [ 2.125448] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.129025] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.132248] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.638446] Initialise system trusted keyrings [ 2.640266] Key type blacklist registered [ 2.642837] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.652359] zbud: loaded [ 2.655170] *** VALIDATE nfs *** [ 2.656530] *** VALIDATE nfs4 *** [ 2.657863] pstore: using deflate compression [ 2.660910] Platform Keyring initialized [ 2.754183] NET: Registered protocol family 38 [ 2.755765] Key type asymmetric registered [ 2.756819] Asymmetric key parser 'x509' registered [ 2.758562] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.761758] io scheduler mq-deadline registered [ 2.762943] io scheduler kyber registered [ 2.765157] io scheduler bfq registered [ 2.769459] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.772643] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.775849] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.779528] ACPI: Power Button [PWRF] [ 2.785403] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.793045] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.806100] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.834417] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.865684] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.870468] Non-volatile memory driver v1.3 [ 2.872128] Linux agpgart interface v0.103 [ 2.902460] virtio_blk virtio1: [vda] 134664 512-byte logical blocks (68.9 MB/65.8 MiB) [ 2.905550] vda: detected capacity change from 0 to 68947968 [ 2.922280] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.925965] vdb: detected capacity change from 0 to 1073741824 [ 2.934253] libphy: Fixed MDIO Bus: probed [ 2.939914] usbcore: registered new interface driver usbserial_generic [ 2.942835] usbserial: USB Serial support registered for generic [ 2.945583] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.950610] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.952601] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.955192] mousedev: PS/2 mouse device common for all mice [ 2.960516] rtc_cmos 00:05: RTC can wake from S4 [ 2.961985] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.964696] rtc_cmos 00:05: registered as rtc0 [ 2.969422] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.970042] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.972406] intel_pstate: CPU model not supported [ 2.978194] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.982851] hid: raw HID events driver (C) Jiri Kosina [ 2.985172] usbcore: registered new interface driver usbhid [ 2.987099] usbhid: USB HID core driver [ 2.988370] drop_monitor: Initializing network drop monitor service [ 2.990379] Initializing XFRM netlink socket [ 2.992592] NET: Registered protocol family 10 [ 2.995915] Segment Routing with IPv6 [ 2.997481] NET: Registered protocol family 17 [ 2.999568] mpls_gso: MPLS GSO support [ 3.005950] RAS: Correctable Errors collector initialized. [ 3.007626] AVX version of gcm_enc/dec engaged. [ 3.009268] AES CTR mode by8 optimization enabled [ 3.091054] sched_clock: Marking stable (3091014262, 0)->(4060742511, -969728249) [ 3.094892] registered taskstats version 1 [ 3.096961] Loading compiled-in X.509 certificates [ 3.098951] zswap: loaded using pool lzo/zbud [ 3.126864] Key type big_key registered [ 3.139270] Key type encrypted registered [ 3.141511] ima: No TPM chip found, activating TPM-bypass! [ 3.144174] ima: Allocated hash algorithm: sha1 [ 3.145902] ima: No architecture policies found [ 3.147659] evm: Initialising EVM extended attributes: [ 3.149556] evm: security.selinux [ 3.150983] evm: security.ima [ 3.152132] evm: security.capability [ 3.153673] evm: HMAC attrs: 0x1 [ 3.156275] rtc_cmos 00:05: setting system clock to 2026-06-09 01:01:11 UTC (1780966871) [ 3.162496] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 3.165899] debug: unmapping init [mem 0xffffffffae582000-0xffffffffae858fff] [ 3.174126] Write protecting the kernel read-only data: 28672k [ 3.177159] debug: unmapping init [mem 0xffffffffacc03000-0xffffffffacdfffff] [ 3.180354] debug: unmapping init [mem 0xffffffffad514000-0xffffffffad5fffff] [ 3.216816] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.224654] systemd[1]: Detected virtualization kvm. [ 3.226354] systemd[1]: Detected architecture x86-64. [ 3.228167] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.253763] systemd[1]: No hostname configured. [ 3.255762] systemd[1]: Set hostname to . [ 3.257897] random: systemd: uninitialized urandom read (16 bytes read) [ 3.260626] systemd[1]: Initializing machine ID from random generator. [ 3.411214] random: systemd: uninitialized urandom read (16 bytes read) [ 3.413625] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.417288] random: systemd: uninitialized urandom read (16 bytes read) [ 3.420546] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.425776] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.106508] device-mapper: uevent: version 1.0.3 [ 4.109145] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.887782] virtio_net virtio0 ens2: renamed from eth0 [ 4.913384] scsi host0: ata_piix [ 4.927423] scsi host1: ata_piix [ 4.929193] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.932040] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.650176] random: fast init done [ 8.703254] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.659765] random: crng init done [ 9.661766] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.097939] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.244981] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.490667] SELinux: Disabled at runtime. [ 11.551729] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.561221] systemd[1]: Detected virtualization kvm. [ 11.563362] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.064195] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.068171] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.073463] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.078302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.082517] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.093183] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.105726] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 12.235514] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.576034] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.927044] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.935456] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.064049] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.084438] EDAC sbridge: Ver: 1.1.2 [ 14.301488] Key type dns_resolver registered [ 14.610172] NFS: Registering the id_resolver key type [ 14.612126] Key type id_resolver registered [ 14.613657] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-client login: [ 46.668533] libcfs: loading out-of-tree module taints kernel. [ 46.732638] Key type ._llcrypt registered [ 46.741463] Key type .llcrypt registered [ 47.002828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.010439] alg: No test for adler32 (adler32-zlib) [ 48.069739] Lustre: Lustre: Build Version: 2.17.52_126_g9f56455 [ 48.427860] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 50.080203] Key type lgssc registered [ 50.951061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 221.445398] Lustre: Mounted lustre-client [ 227.910064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.175991] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.264216] Lustre: lustre-OST0000-osc-ffff8d7dc3db9000: disconnect after 23s idle [ 249.699870] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing yml_node [ 253.551887] Lustre: DEBUG MARKER: Client: 2.17.52.126 [ 255.966961] Lustre: DEBUG MARKER: MDS: 2.17.52.126 [ 258.662874] Lustre: DEBUG MARKER: OSS: 2.17.52.126 [ 260.151552] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 8 21:05:27 EDT 2026 [ 280.665671] LustreError: 8546:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8d7dc3db9000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 280.767062] Lustre: Unmounted lustre-client [ 341.220797] Key type lgssc unregistered [ 341.607889] LNet: 9199:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.617743] LNetError: 9199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.639858] LNet: Removed LNI 192.168.204.23@tcp [ 342.648590] Key type .llcrypt unregistered [ 342.655695] Key type ._llcrypt unregistered [ 385.384134] Key type ._llcrypt registered [ 385.385593] Key type .llcrypt registered [ 385.816551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 385.839320] alg: No test for adler32 (adler32-zlib) [ 386.925253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 387.028378] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 387.057435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 393.277054] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 393.407844] LNet: 10128:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.432902] LNetError: 10128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.456084] LNet: Removed LNI 192.168.204.23@tcp [ 394.256332] Key type .llcrypt unregistered [ 394.264265] Key type ._llcrypt unregistered [ 402.736651] Key type ._llcrypt registered [ 402.743822] Key type .llcrypt registered [ 404.588682] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:07:50 (1780967270) [ 405.051183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 405.072891] alg: No test for adler32 (adler32-zlib) [ 406.041523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.137465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.158185] LNet: 10548:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.178163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.228149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.253428] LNet: 10555:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 415.745652] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:08:02 (1780967282) [ 415.883434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.967143] LNet: 11135:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.016427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.200440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 424.758439] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:08:11 (1780967291) [ 424.876144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.926548] LNet: 11728:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.971947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.108035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 426.387203] hrtimer: interrupt took 11171132 ns [ 433.381472] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:08:20 (1780967300) [ 433.803124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.845381] LNet: 12331:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.876548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.984661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 442.956827] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:08:29 (1780967309) [ 443.087384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.164584] LNet: 12916:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.226631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.433045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 444.398731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 453.583747] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:08:40 (1780967320) [ 453.668192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.698378] LNet: 13514:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.740522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.888202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 454.422789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 455.048365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 463.050435] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:08:49 (1780967329) [ 463.164507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.258338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.389915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 471.028376] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:08:57 (1780967337) [ 471.272353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.296335] LNet: 14719:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.302288] LNet: 14719:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 471.328123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.404727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 471.437774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 472.001839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 472.043223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 472.632853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 472.671657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 473.238873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 473.276786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 473.765829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 473.819488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 474.394694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 474.438432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 474.969808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 475.038457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 475.633301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 475.679738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 476.271765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 476.323801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 476.848524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 476.890652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 477.440792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 477.537572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 478.216791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 478.261963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 478.869616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 478.937729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 486.972884] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:09:13 (1780967353) [ 487.073154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.159080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.301499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 487.335364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 495.028989] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:09:21 (1780967361) [ 495.286515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.387675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.489390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 495.547460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 503.729615] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:09:30 (1780967370) [ 503.863789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.921794] LNet: 16593:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.930087] LNet: 16593:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 503.965857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.089131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 504.159511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 512.871915] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:09:39 (1780967379) [ 512.974105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.022427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.117805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 513.152878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 521.734971] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:09:48 (1780967388) [ 521.983556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.015674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.073840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 522.118367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 529.683788] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:09:56 (1780967396) [ 529.802222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.894446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.032869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 530.072726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 538.152344] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:10:04 (1780967404) [ 538.302220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.369211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.544374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.290236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 548.072226] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:10:14 (1780967414) [ 548.456519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.518675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.795636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.149746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 558.180755] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:10:24 (1780967424) [ 558.295333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.411378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.697285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.127769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 568.685200] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:10:35 (1780967435) [ 568.800691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.882267] LNet: 20826:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.889878] LNet: 20826:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 568.939082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.962682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.507964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 579.456870] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:10:45 (1780967445) [ 579.543164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.635724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.782220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.852889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 587.364128] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:10:54 (1780967454) [ 587.867504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.948785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.094216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 588.182774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 596.868362] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:11:03 (1780967463) [ 597.099765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.181410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.257799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 604.681159] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:11:11 (1780967471) [ 604.990749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.057596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.210307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 613.431113] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:11:20 (1780967480) [ 613.786533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.830402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.953304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 614.672977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 622.582439] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:11:29 (1780967489) [ 622.733761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.789109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.958969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 623.066270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 631.689797] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:11:38 (1780967498) [ 631.791624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.844102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.975559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 632.039358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 639.464940] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:11:46 (1780967506) [ 639.570109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.615526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.715835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 639.783710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 647.325504] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:11:53 (1780967513) [ 647.445409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.487303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.609259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 647.706383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 656.131755] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:12:02 (1780967522) [ 656.305865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.385374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.468536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 656.564879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 656.597448] LNetError: 26886:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 656.689428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 656.742260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 664.698680] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:12:11 (1780967531) [ 664.905701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.025048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.153969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WhkV0pybFO [ 665.263460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WhkV0pybFO [ 665.338477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WhkV0pybFO [ 672.135726] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:12:19 (1780967539) [ 672.490655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.543545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.634854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 672.680246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 672.771830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 681.686903] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:12:27 (1780967547) [ 681.871414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.933344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.046945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.227356] LNet: Message forwarding will be enabled in 2s [ 688.353881] LNet: Message forwarding enabled [ 688.789741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 688.817859] LNet: Message forwarding will stop in 2s to 6s [ 693.450925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 694.247466] LNet: Message forwarding disabled [ 696.068043] LNet: Message forwarding will be enabled in 2s [ 699.360106] LNet: Message forwarding enabled [ 704.321346] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:12:51 (1780967571) [ 704.587370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.621362] LNet: 29364:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.631530] LNet: 29364:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 705.877676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.968207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 712.898531] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:12:59 (1780967579) [ 713.110698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.181406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.266836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 713.322166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 713.371673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 713.415464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 713.458627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 713.506846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 713.557553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 713.597513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 713.636242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 713.694249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 713.739574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 713.779987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 713.829459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 713.868759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 713.903431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 713.946697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 713.982674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 714.022112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 714.064394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 714.081129] LNetError: 29996:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 720.815908] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:13:07 (1780967587) [ 720.973910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.039355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.158244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.878783] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:13:14 (1780967594) [ 730.192398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg423-server [ 732.891420] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:13:19 (1780967599) [ 732.980823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.040150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.153945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.227416] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 734.828703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[24-28]@tcp [ 734.849443] LNet: 31737:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 734.909227] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 734.936703] LNetError: 31521:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.24@tcp has gone from up to down [ 735.915080] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 735.926987] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 736.937835] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 736.958499] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 738.986709] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.26@tcp is being used as a gateway but routing feature is not turned on [ 739.007166] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 743.078911] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 743.092650] LNetError: 31521:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 743.177399] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:13:29 (1780967609) [ 745.732439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg423-server [ 747.786923] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:13:34 (1780967614) [ 748.070409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.099665] LNetError: 32686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.123277] LNet: Removed LNI 192.168.204.23@tcp [ 748.182263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.250413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.286939] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 749.172754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[24-29/2]@tcp [ 749.208846] LNet: 32803:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 749.221769] LNet: 32803:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 749.245478] LNetError: 32690:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.24@tcp has gone from up to down [ 749.272746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[24-29/2]@tcp [ 749.276641] LNetError: 32690:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 757.273319] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:13:43 (1780967623) [ 757.385216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.428345] LNetError: 33398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.471367] LNet: Removed LNI 192.168.204.23@tcp [ 757.505216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.646248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 757.717712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.762685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.845654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 757.914518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.942100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.987342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 758.072557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.120550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.195329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 758.342466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.387671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.436623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 758.550103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.603892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.683550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 758.787278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.843576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.920647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 759.026983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.092966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.169838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.272562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.304922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.353414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 759.445813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.477607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.558647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.690498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.777747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.863963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 759.931353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.992317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.092504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 766.825236] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:13:53 (1780967633) [ 768.820583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg423-server [ 770.744967] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:13:57 (1780967637) [ 773.210336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg423-server [ 774.955809] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:14:01 (1780967641) [ 775.859504] Key type .llcrypt unregistered [ 775.865148] Key type ._llcrypt unregistered [ 782.098308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.237787] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.244131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.253653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.003576] Key type ._llcrypt registered [ 784.009370] Key type .llcrypt registered [ 784.313674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.342882] alg: No test for adler32 (adler32-zlib) [ 785.353889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.477243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.538046] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 786.674950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 786.708849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.170662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 790.201198] LNet: Removed LNI 10.1.2.3@tcp [ 790.372574] LNet: 35541:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.380802] LNetError: 35541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.395761] LNet: Removed LNI 192.168.204.23@tcp [ 791.413778] Key type .llcrypt unregistered [ 791.422915] Key type ._llcrypt unregistered [ 804.823501] Key type ._llcrypt registered [ 804.828031] Key type .llcrypt registered [ 806.447215] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:14:33 (1780967673) [ 807.009385] Key type .llcrypt unregistered [ 807.011360] Key type ._llcrypt unregistered [ 813.713105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.865412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.870689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.875708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.612716] Key type ._llcrypt registered [ 816.618123] Key type .llcrypt registered [ 816.950578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.994266] alg: No test for adler32 (adler32-zlib) [ 818.036442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.092550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.112984] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 819.000097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 819.026185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.117584] LNet: 37048:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.129813] LNetError: 37048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.137296] LNet: Removed LNI 192.168.204.23@tcp [ 822.147249] LNet: Removed LNI 10.1.2.3@tcp [ 822.838457] Key type .llcrypt unregistered [ 822.842165] Key type ._llcrypt unregistered [ 835.532807] Key type ._llcrypt registered [ 835.535264] Key type .llcrypt registered [ 837.185735] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:15:03 (1780967703) [ 837.579518] Key type .llcrypt unregistered [ 837.581943] Key type ._llcrypt unregistered [ 843.147180] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.403072] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.424091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.446581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.181838] Key type ._llcrypt registered [ 846.183981] Key type .llcrypt registered [ 846.411315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.427110] alg: No test for adler32 (adler32-zlib) [ 847.366523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.448193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 847.476579] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 849.604324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 849.623885] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.646792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 851.671979] LNet: Removed LNI 10.1.2.3@tcp [ 851.695472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 851.749460] LNet: Removed LNI 10.1.2.31@tcp [ 851.766520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 851.798444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.590284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 852.605165] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 853.566699] LNet: 38981:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.574506] LNetError: 38981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.579246] LNet: Removed LNI 10.1.2.3@tcp [ 853.591139] LNet: Removed LNI 10.1.2.31@tcp [ 854.210544] Key type .llcrypt unregistered [ 854.212402] Key type ._llcrypt unregistered [ 865.585399] Key type ._llcrypt registered [ 865.591385] Key type .llcrypt registered [ 866.780335] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:15:33 (1780967733) [ 867.169927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.182879] alg: No test for adler32 (adler32-zlib) [ 868.072750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.153286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.170176] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 869.021972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.065099] LNet: 39944:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.080491] LNetError: 39944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.130765] LNet: Removed LNI 192.168.204.23@tcp [ 869.174461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.231194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.23@tcp [ 869.253858] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 870.345164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.23@tcp [ 870.394104] LNet: Removed LNI 192.168.204.23@tcp [ 870.504645] LNet: 40079:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.364681] Key type .llcrypt unregistered [ 871.371932] Key type ._llcrypt unregistered [ 885.720857] Key type ._llcrypt registered [ 885.722751] Key type .llcrypt registered [ 887.326617] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:15:53 (1780967753) [ 890.120289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg423-server [ 891.932114] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:15:58 (1780967758) [ 892.877412] Key type .llcrypt unregistered [ 892.882516] Key type ._llcrypt unregistered [ 898.371573] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.471075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 898.476241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 898.484440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.060612] Key type ._llcrypt registered [ 900.073132] Key type .llcrypt registered [ 900.296037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.314582] alg: No test for adler32 (adler32-zlib) [ 901.278548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.347931] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 902.151952] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 904.597586] LNet: 41937:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.614301] LNetError: 41937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.621969] LNet: Removed LNI 192.168.204.23@tcp1 [ 904.643032] LNet: Removed LNI 10.1.2.3@tcp2 [ 905.532369] Key type .llcrypt unregistered [ 905.534178] Key type ._llcrypt unregistered [ 919.584413] Key type ._llcrypt registered [ 919.587753] Key type .llcrypt registered [ 920.742494] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:16:27 (1780967787) [ 921.699584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.715229] alg: No test for adler32 (adler32-zlib) [ 922.766823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.820353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.849368] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 923.783760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.795338] LNet: 42916:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.798936] LNetError: 42916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.813052] LNet: Removed LNI 192.168.204.23@tcp [ 923.832807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.916128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 923.946273] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 932.185780] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:16:39 (1780967799) [ 932.293745] LNet: 43509:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.299947] LNetError: 43509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.317646] LNet: Removed LNI 192.168.204.23@tcp [ 933.165943] Key type .llcrypt unregistered [ 933.173114] Key type ._llcrypt unregistered [ 938.989737] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.263064] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.266309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.275550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 941.771629] Key type ._llcrypt registered [ 941.774311] Key type .llcrypt registered [ 942.016638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.053343] alg: No test for adler32 (adler32-zlib) [ 942.914440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.980736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.023340] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 943.772319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 943.800984] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.650183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.668905] LNet: 44261:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.675676] LNetError: 44261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.690600] LNet: Removed LNI 192.168.204.23@tcp [ 946.699253] LNet: Removed LNI 10.1.2.3@tcp [ 946.721663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.795422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 946.842062] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 946.846810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.501703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.540534] LNet: 44288:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.547558] LNetError: 44288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.553897] LNetError: 44288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 947.561623] LNet: Removed LNI 192.168.204.23@tcp [ 947.588652] LNet: Removed LNI 10.1.2.3@tcp [ 947.669227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.747891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 947.775932] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 947.785611] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.239679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 948.260783] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 949.560565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.581665] LNet: 44450:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.591772] LNetError: 44450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.600614] LNetError: 44450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 949.610536] LNet: Removed LNI 192.168.204.23@tcp [ 949.619654] LNet: Removed LNI 10.1.2.3@tcp [ 949.632256] LNet: Removed LNI 192.168.204.23@tcp2 [ 949.656264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.737785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 949.770639] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 949.776032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.783061] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 957.212167] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:17:04 (1780967824) [ 957.307830] LNet: 45045:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.318618] LNetError: 45045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.328944] LNetError: 45045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 957.336088] LNet: Removed LNI 192.168.204.23@tcp [ 957.337666] LNet: Removed LNI 10.1.2.3@tcp [ 957.357297] LNet: Removed LNI 192.168.204.23@tcp2 [ 957.959862] Key type .llcrypt unregistered [ 957.962487] Key type ._llcrypt unregistered [ 964.164986] Key type ._llcrypt registered [ 964.175085] Key type .llcrypt registered [ 964.384809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.413490] alg: No test for adler32 (adler32-zlib) [ 965.346658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.446584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.490906] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 966.324675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.344234] LNet: 45452:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.350735] LNetError: 45452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.375294] LNet: Removed LNI 192.168.204.23@tcp [ 966.392526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.463953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 966.498544] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 967.021943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.054418] LNet: 45482:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.061147] LNetError: 45482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.080519] LNet: Removed LNI 192.168.204.23@tcp [ 967.108356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.195294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 967.219086] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 967.702485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.751766] LNet: Removed LNI 192.168.204.23@tcp [ 967.770910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.820307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 967.837835] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 968.289867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.446485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.460825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.475861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.892660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.918608] LNet: 45585:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.926279] LNet: 45585:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 970.935788] LNetError: 45585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.942740] LNetError: 45585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 970.972658] LNet: Removed LNI 192.168.204.23@tcp [ 971.016753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.097753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 971.124566] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 978.759180] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:17:25 (1780967845) [ 979.054663] LNet: 46191:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.065913] LNetError: 46191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.094964] LNet: Removed LNI 192.168.204.23@tcp [ 979.910766] Key type .llcrypt unregistered [ 979.914252] Key type ._llcrypt unregistered [ 986.710278] Key type ._llcrypt registered [ 986.713133] Key type .llcrypt registered [ 986.941583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 986.970733] alg: No test for adler32 (adler32-zlib) [ 987.995570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.087357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.128064] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 989.238797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.278366] LNet: 46591:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.288827] LNetError: 46591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.310688] LNet: Removed LNI 192.168.204.23@tcp [ 989.325655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.421592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 989.450564] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 996.698946] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:17:43 (1780967863) [ 997.078445] LNet: 47191:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.089548] LNetError: 47191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.121616] LNet: Removed LNI 192.168.204.23@tcp [ 997.766777] Key type .llcrypt unregistered [ 997.770504] Key type ._llcrypt unregistered [ 1004.348781] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1004.646718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1004.653955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1004.656895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1007.278990] Key type ._llcrypt registered [ 1007.282509] Key type .llcrypt registered [ 1007.508037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.531516] alg: No test for adler32 (adler32-zlib) [ 1008.499474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.606252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.638070] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1009.694316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.729581] LNet: 47638:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.735130] LNetError: 47638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.753949] LNet: Removed LNI 192.168.204.23@tcp [ 1009.808972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.868754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1009.898876] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1017.786237] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:18:04 (1780967884) [ 1018.078946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.107468] LNet: 48245:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.117580] LNetError: 48245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.141674] LNet: Removed LNI 192.168.204.23@tcp [ 1018.170297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.297258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1025.191784] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:18:12 (1780967892) [ 1025.455296] LNet: 48835:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.169635] Key type .llcrypt unregistered [ 1026.177139] Key type ._llcrypt unregistered [ 1032.881195] Key type ._llcrypt registered [ 1032.888712] Key type .llcrypt registered [ 1033.092225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.106390] alg: No test for adler32 (adler32-zlib) [ 1034.024558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.106091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.145385] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1034.962031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.976604] LNet: 49234:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.982417] LNetError: 49234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.993849] LNet: Removed LNI 192.168.204.23@tcp [ 1035.022518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.150516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1035.195344] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1044.733464] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:18:31 (1780967911) [ 1044.865063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.897619] LNet: 49829:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.917606] LNetError: 49829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.947466] LNet: Removed LNI 192.168.204.23@tcp [ 1044.985170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.095835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1052.032646] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:18:38 (1780967918) [ 1053.856552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1055.870500] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:18:42 (1780967922) [ 1058.163988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1060.807452] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:18:47 (1780967927) [ 1063.069153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1064.977898] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:18:51 (1780967931) [ 1066.976903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1069.097658] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:18:55 (1780967935) [ 1070.698166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1072.406265] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:18:59 (1780967939) [ 1072.747845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.768836] LNet: 51918:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.790494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.886535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1072.918064] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1073.785498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.23 [ 1073.808467] LNet: 52051:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1073.819941] LNet: Removed LNI 192.168.204.23@tcp [ 1081.087229] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:19:07 (1780967947) [ 1081.185461] LNet: 52624:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.193975] LNetError: 52624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.228951] LNet: Removed LNI 192.168.204.23@tcp [ 1082.179515] Key type .llcrypt unregistered [ 1082.181406] Key type ._llcrypt unregistered [ 1088.720846] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1088.999768] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1089.005327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1089.014871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.789936] Key type ._llcrypt registered [ 1091.792851] Key type .llcrypt registered [ 1091.982264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.998791] alg: No test for adler32 (adler32-zlib) [ 1093.418963] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1100.334078] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:19:27 (1780967967) [ 1100.824250] LNet: 53547:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.830247] LNetError: 53547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.850442] LNet: Removed LNI 10.1.2.3@tcp [ 1101.595685] Key type .llcrypt unregistered [ 1101.601252] Key type ._llcrypt unregistered [ 1109.203685] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1109.424761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1109.429368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1109.434780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1112.261558] Key type ._llcrypt registered [ 1112.264729] Key type .llcrypt registered [ 1112.467340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.497597] alg: No test for adler32 (adler32-zlib) [ 1113.475822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1120.725526] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:19:47 (1780967987) [ 1120.957561] LNet: 54452:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.969659] LNetError: 54452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1120.983452] LNet: Removed LNI 10.1.2.3@tcp [ 1121.705912] Key type .llcrypt unregistered [ 1121.708940] Key type ._llcrypt unregistered [ 1128.248811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1128.435608] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1128.438288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1128.440356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1131.211081] Key type ._llcrypt registered [ 1131.212744] Key type .llcrypt registered [ 1131.467232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.485871] alg: No test for adler32 (adler32-zlib) [ 1132.449725] LNet: 54753:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1132.468595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1140.429141] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:20:06 (1780968006) [ 1140.601130] LNet: 55345:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.625250] LNetError: 55345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1140.659785] LNet: Removed LNI 10.1.2.3@tcp [ 1141.525875] Key type .llcrypt unregistered [ 1141.527816] Key type ._llcrypt unregistered [ 1148.453484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1148.743365] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1148.756688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1148.766086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1151.591294] Key type ._llcrypt registered [ 1151.594276] Key type .llcrypt registered [ 1151.828562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1151.842102] alg: No test for adler32 (adler32-zlib) [ 1152.753045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.813360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1161.923827] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:20:28 (1780968028) [ 1162.528475] LNet: 56595:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.550327] LNetError: 56595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1162.565343] LNet: Removed LNI 10.1.2.3@tcp [ 1163.189608] Key type .llcrypt unregistered [ 1163.192466] Key type ._llcrypt unregistered [ 1170.785833] Key type ._llcrypt registered [ 1170.789319] Key type .llcrypt registered [ 1171.000340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.018776] alg: No test for adler32 (adler32-zlib) [ 1172.051746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.109626] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1181.508863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1182.327768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_interrupt -r 1 [ 1182.373812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_interrupt -r 1 [ 1182.403566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1182.418169] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1182.662428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.704603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1182.739662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.780884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.816376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.879434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.964780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.031460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.065622] LNet: 57035:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.104758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.137706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.175814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.218854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.282109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.333415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.370489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.961576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_dropped -r 1 [ 1183.999755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_dropped -r 1 [ 1184.044737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1184.072509] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1184.386529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.416798] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1184.448983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1184.493800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.557386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.642275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.712529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.767115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.822524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.848527] LNet: 57144:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.888409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.942672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.995848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.058635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.111448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.193522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.264418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.084385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_aborted -r 1 [ 1186.133597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_aborted -r 1 [ 1186.204994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1186.244589] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1186.535719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.559263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1186.589768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.641273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.690855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.743745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.783769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.830946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.861170] LNet: 57253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.890676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.943751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.990307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.028342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.076262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.113975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.167072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1187.924528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_no_route -r 1 [ 1187.998540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_no_route -r 1 [ 1188.061530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1188.334974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.403642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1188.453850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.481246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.512179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.550650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.611040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.670964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.704625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.767709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.828805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.881994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.976833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.004500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.057655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.723194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_timeout -r 1 [ 1189.763184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_timeout -r 1 [ 1189.798342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1189.829668] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1189.834982] LNet: Skipped 1 previous similar message [ 1190.047178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.087244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1190.129642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.173776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.219048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.247716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.289593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.337609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.362213] LNet: 57472:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.377158] LNet: 57472:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1190.402497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.445663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.481382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.532355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.578534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.629839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.689200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.231732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_error -r 1 [ 1191.266083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_error -r 1 [ 1191.297384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1191.574561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.613785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1191.640486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.669439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.695385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.750036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.803572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.829667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.880071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.911355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.964987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.002338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.033105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.088255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.128800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.419151] LNet: 57622:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.439591] LNetError: 57622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.464581] LNet: Removed LNI 192.168.204.23@tcp [ 1193.108128] Key type .llcrypt unregistered [ 1193.110622] Key type ._llcrypt unregistered [ 1213.158759] Key type ._llcrypt registered [ 1213.161457] Key type .llcrypt registered [ 1214.449129] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:21:21 (1780968081) [ 1215.165117] Key type .llcrypt unregistered [ 1215.169661] Key type ._llcrypt unregistered [ 1221.392735] Key type ._llcrypt registered [ 1221.395585] Key type .llcrypt registered [ 1221.662561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1221.691613] alg: No test for adler32 (adler32-zlib) [ 1222.623876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1222.659875] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1229.916774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1240.718273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.762833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1240.778659] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1242.198460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_interrupt -r 1 [ 1242.232712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_interrupt -r 1 [ 1242.276875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_interrupt -r 1 [ 1242.315651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_interrupt -r 1 [ 1242.376651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_interrupt -r 1 [ 1242.441979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_interrupt -r 1 [ 1242.496908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_interrupt -r 1 [ 1242.538896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_interrupt -r 1 [ 1242.581910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1242.600089] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1242.609433] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1242.798390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.833695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1242.877800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.904671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.933639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.973134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.008511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.049339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.075495] LNet: 59118:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.097351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.145712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.176158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.211345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.237565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.266769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.299143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.793977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_dropped -r 1 [ 1243.832495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_dropped -r 1 [ 1243.880747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_dropped -r 1 [ 1243.918726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_dropped -r 1 [ 1243.967621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_dropped -r 1 [ 1243.998845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_dropped -r 1 [ 1244.038307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_dropped -r 1 [ 1244.071224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_dropped -r 1 [ 1244.113407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1244.149754] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1244.155274] LNet: Skipped 2 previous similar messages [ 1244.342693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.368299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1244.401887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.431436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.457902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.506120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.550325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.606086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.633941] LNet: 59239:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.644739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.686708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.735771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.764707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.800729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.843868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.891750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.509600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_aborted -r 1 [ 1245.574526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_aborted -r 1 [ 1245.635197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_aborted -r 1 [ 1245.685711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_aborted -r 1 [ 1245.732927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_aborted -r 1 [ 1245.781570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_aborted -r 1 [ 1245.812581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_aborted -r 1 [ 1245.851629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_aborted -r 1 [ 1245.898730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1245.944508] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1245.953147] LNet: Skipped 2 previous similar messages [ 1246.206386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.277696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.321178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.378527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.410952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.441936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.472922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.521480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.556398] LNet: 59360:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.577915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.624767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.662962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.700535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.759503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.828803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.872882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.377271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_no_route -r 1 [ 1247.414359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_no_route -r 1 [ 1247.449885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_no_route -r 1 [ 1247.478919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_no_route -r 1 [ 1247.512630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_no_route -r 1 [ 1247.586988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_no_route -r 1 [ 1247.635092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_no_route -r 1 [ 1247.671150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_no_route -r 1 [ 1247.718692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1247.931477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.960791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1247.995811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.033583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.090389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.134697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.169694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.205417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.241314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.279216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.315916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.342830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.391550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.441520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.473360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1249.101883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_timeout -r 1 [ 1249.143254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_timeout -r 1 [ 1249.181386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_timeout -r 1 [ 1249.216692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_timeout -r 1 [ 1249.252206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_timeout -r 1 [ 1249.285398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_timeout -r 1 [ 1249.311226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_timeout -r 1 [ 1249.339851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_timeout -r 1 [ 1249.368091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1249.389315] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1249.398479] LNet: Skipped 5 previous similar messages [ 1249.600349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.638465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1249.667327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.696379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.742902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.781087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.813591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.850456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.871462] LNet: 59603:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.875749] LNet: 59603:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1249.894762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.950087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.992303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.035389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.072631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.115129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.160256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1250.665270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_error -r 1 [ 1250.721609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_error -r 1 [ 1250.775949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_error -r 1 [ 1250.846856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_error -r 1 [ 1250.909569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_error -r 1 [ 1250.942881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_error -r 1 [ 1250.994898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_error -r 1 [ 1251.052950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_error -r 1 [ 1251.100603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1251.387531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.423255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1251.457226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.491965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.523426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.550857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.575450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.612404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.656376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.701675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.732693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.780287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.824348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.889424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.930489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.984813] LNet: 59800:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.991544] LNetError: 59800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.000755] LNet: Removed LNI 192.168.204.23@tcp [ 1257.017258] LNet: Removed LNI 192.168.204.23@tcp1 [ 1257.375820] Key type .llcrypt unregistered [ 1257.379801] Key type ._llcrypt unregistered [ 1273.558890] Key type ._llcrypt registered [ 1273.561297] Key type .llcrypt registered [ 1274.993683] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:22:21 (1780968141) [ 1275.875266] Key type .llcrypt unregistered [ 1275.878773] Key type ._llcrypt unregistered [ 1282.542872] Key type ._llcrypt registered [ 1282.549334] Key type .llcrypt registered [ 1282.765849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.788159] alg: No test for adler32 (adler32-zlib) [ 1283.696326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.727718] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1293.092382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1294.648526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_dropped -r 1 [ 1294.735522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_dropped -r 1 [ 1294.830564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1294.883723] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1295.282646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.330971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1295.433951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.535145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.642116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.778219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.872836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.909137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.953826] LNet: 61062:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.990658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.051540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.123398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.203681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.269518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.322385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.374118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1297.236696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_error -r 1 [ 1297.327188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_error -r 1 [ 1297.393252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1297.411257] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1297.789220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.832619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1297.878515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.912651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.947489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.038040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.145592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.217522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.257613] LNet: 61172:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.292499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.362753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.449188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.549272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.593375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.642950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.694894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.729268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_timeout -r 1 [ 1299.767392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_timeout -r 1 [ 1299.797948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1299.840222] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1300.259717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.351681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1300.407533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.462702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.515855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.558720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.622817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1300.718682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1300.741562] LNet: 61281:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1300.761668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1300.786903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1300.816695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.844955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.869647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.893571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1300.920322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.118922] LNet: 61322:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.123513] LNetError: 61322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.134439] LNet: Removed LNI 192.168.204.23@tcp [ 1301.513678] Key type .llcrypt unregistered [ 1301.520267] Key type ._llcrypt unregistered [ 1324.294777] Key type ._llcrypt registered [ 1324.297876] Key type .llcrypt registered [ 1325.967721] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:23:12 (1780968192) [ 1326.805431] Key type .llcrypt unregistered [ 1326.813710] Key type ._llcrypt unregistered [ 1333.911789] Key type ._llcrypt registered [ 1333.913948] Key type .llcrypt registered [ 1334.062713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1334.077367] alg: No test for adler32 (adler32-zlib) [ 1335.043617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1335.065152] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1343.466859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1351.776533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.831839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.882208] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1353.502906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_dropped -r 1 [ 1353.559287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e remote_dropped -r 1 [ 1353.616974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_dropped -r 1 [ 1353.683034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e remote_dropped -r 1 [ 1353.731042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e remote_dropped -r 1 [ 1353.799893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e remote_dropped -r 1 [ 1353.858987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e remote_dropped -r 1 [ 1353.947263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e remote_dropped -r 1 [ 1354.040557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1354.090705] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1354.118330] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.123@tcp [ 1354.516989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.587801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1354.647887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.738246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.776379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.832401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.893636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.942754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.959833] LNet: 62805:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1354.980522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.036933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.091296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.145079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.170053] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1355.188905] LNet: Skipped 3 previous similar messages [ 1355.220189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.307075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.371759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1355.455956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.615877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_error -r 1 [ 1356.773891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e remote_error -r 1 [ 1356.846907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_error -r 1 [ 1356.911183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e remote_error -r 1 [ 1356.984352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e remote_error -r 1 [ 1357.056793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e remote_error -r 1 [ 1357.146961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e remote_error -r 1 [ 1357.308718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e remote_error -r 1 [ 1357.353314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1357.382988] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1357.732244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.801822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1357.857245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.923401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.973995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.050317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.124156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.211201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.236216] LNet: 62929:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.269477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.331196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.414898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.536200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.629382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.746408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.808915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.875237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.831877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_timeout -r 1 [ 1359.883700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e remote_timeout -r 1 [ 1359.952782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_timeout -r 1 [ 1360.030878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e remote_timeout -r 1 [ 1360.097080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e remote_timeout -r 1 [ 1360.185858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e remote_timeout -r 1 [ 1360.296315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e remote_timeout -r 1 [ 1360.416189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e remote_timeout -r 1 [ 1360.466566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.123@tcp [ 1360.504971] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1360.965709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.019429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1361.132877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.181354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.253975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.339695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.444291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1361.508707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1361.527087] LNet: 63052:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1361.542942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1361.562635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1361.606623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.634674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1361.686318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.777475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.854132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.958762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.165381] LNet: 63130:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.175030] LNetError: 63130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.185774] LNet: Removed LNI 192.168.204.23@tcp [ 1367.207356] LNet: Removed LNI 192.168.204.23@tcp1 [ 1367.893817] Key type .llcrypt unregistered [ 1367.902344] Key type ._llcrypt unregistered [ 1385.113234] Key type ._llcrypt registered [ 1385.114842] Key type .llcrypt registered [ 1386.859603] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:24:13 (1780968253) [ 1387.598427] Key type .llcrypt unregistered [ 1387.605703] Key type ._llcrypt unregistered [ 1393.821068] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1393.999476] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1394.009898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1394.012756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1396.628504] Key type ._llcrypt registered [ 1396.631940] Key type .llcrypt registered [ 1396.791511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.811591] alg: No test for adler32 (adler32-zlib) [ 1397.743772] LNet: 64219:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1397.761301] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1397.821309] LNet: 64233:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.839408] LNetError: 64233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.855272] LNet: Removed LNI 192.168.204.23@tcp [ 1398.617562] Key type .llcrypt unregistered [ 1398.627444] Key type ._llcrypt unregistered [ 1404.554776] Key type ._llcrypt registered [ 1404.559747] Key type .llcrypt registered [ 1404.813672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1404.839292] alg: No test for adler32 (adler32-zlib) [ 1405.859752] LNet: 64495:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1405.894540] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1405.898677] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1405.944521] LNet: 64509:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.951832] LNetError: 64509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.958835] LNet: Removed LNI 192.168.204.23@tcp [ 1405.974417] LNet: Removed LNI 10.1.2.3@tcp1 [ 1406.615511] Key type .llcrypt unregistered [ 1406.620077] Key type ._llcrypt unregistered [ 1412.977189] Key type ._llcrypt registered [ 1412.980379] Key type .llcrypt registered [ 1413.152221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1413.180716] alg: No test for adler32 (adler32-zlib) [ 1414.116164] LNet: 64769:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1414.153687] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1414.159739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1414.218688] LNet: 64783:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.230159] LNetError: 64783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.244273] LNet: Removed LNI 192.168.204.23@tcp [ 1414.256518] LNet: Removed LNI 10.1.2.3@tcp [ 1414.901081] Key type .llcrypt unregistered [ 1414.909297] Key type ._llcrypt unregistered [ 1420.891432] Key type ._llcrypt registered [ 1420.896095] Key type .llcrypt registered [ 1421.103370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.125569] alg: No test for adler32 (adler32-zlib) [ 1422.071624] LNet: 65044:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1422.093298] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1422.108481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1422.157637] LNet: 65058:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.165870] LNetError: 65058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.172850] LNet: Removed LNI 192.168.204.23@tcp [ 1422.185175] LNet: Removed LNI 10.1.2.3@tcp [ 1422.967470] Key type .llcrypt unregistered [ 1422.973793] Key type ._llcrypt unregistered [ 1429.900808] Key type ._llcrypt registered [ 1429.912754] Key type .llcrypt registered [ 1430.174540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.198512] alg: No test for adler32 (adler32-zlib) [ 1431.213862] LNet: 65324:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1431.253175] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1431.262796] LNetError: 65324:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1431.273652] LNetError: Error -100 starting up LNI tcp [ 1431.285192] LNet: Removed LNI 192.168.204.23@tcp [ 1431.913471] Key type .llcrypt unregistered [ 1431.920428] Key type ._llcrypt unregistered [ 1442.808809] Key type ._llcrypt registered [ 1442.810965] Key type .llcrypt registered [ 1444.271532] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:25:11 (1780968311) [ 1445.086639] Key type .llcrypt unregistered [ 1445.090629] Key type ._llcrypt unregistered [ 1450.832340] Key type ._llcrypt registered [ 1450.839759] Key type .llcrypt registered [ 1451.180510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1451.208515] alg: No test for adler32 (adler32-zlib) [ 1452.165433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1452.198725] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1459.417155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1460.328417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e network_timeout -r 1 [ 1460.389720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e network_timeout -r 1 [ 1460.427494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1460.449801] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1460.675586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1460.708771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1460.755618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1460.805024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1460.843243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1460.884458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1460.924941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1460.959415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1460.990432] LNet: 66592:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1461.023288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1461.054151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1461.099350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.145125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1461.204553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1461.279878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1461.599594] LNet: 66632:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.623175] LNetError: 66632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.660348] LNet: Removed LNI 192.168.204.23@tcp [ 1462.064823] Key type .llcrypt unregistered [ 1462.066873] Key type ._llcrypt unregistered [ 1479.945546] Key type ._llcrypt registered [ 1479.948505] Key type .llcrypt registered [ 1480.126530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1480.151717] alg: No test for adler32 (adler32-zlib) [ 1481.071944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1481.093768] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1488.837687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1497.087720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.124880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.145761] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1498.836482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e network_timeout -r 1 [ 1498.904453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e network_timeout -r 1 [ 1498.966728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e network_timeout -r 1 [ 1499.015474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e network_timeout -r 1 [ 1499.094160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e network_timeout -r 1 [ 1499.151512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e network_timeout -r 1 [ 1499.247515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e network_timeout -r 1 [ 1499.316054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e network_timeout -r 1 [ 1499.386821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1499.455286] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1499.556597] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1499.561577] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.123@tcp [ 1499.828289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1499.885484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1499.943700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1500.006243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1500.046727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1500.129089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1500.198919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1500.263107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1500.300972] LNet: 67516:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1500.330852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1500.421748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1500.486544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1500.544458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1500.577866] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1500.604324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1500.695772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1505.852328] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1505.864109] LNet: Skipped 1 previous similar message [ 1506.289782] LNet: 67590:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.295837] LNetError: 67590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.360635] LNet: Removed LNI 192.168.204.23@tcp [ 1507.367853] LNetError: 67590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.384040] LNet: Removed LNI 192.168.204.23@tcp1 [ 1508.047294] Key type .llcrypt unregistered [ 1508.051106] Key type ._llcrypt unregistered [ 1524.075445] Key type ._llcrypt registered [ 1524.077371] Key type .llcrypt registered [ 1525.445243] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:26:32 (1780968392) [ 1525.973246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.990655] alg: No test for adler32 (adler32-zlib) [ 1526.886640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.952102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1526.988828] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1527.756083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1527.769094] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1528.445787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 1528.527556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1528.574952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1528.608453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1528.643453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1528.680672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1528.719936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.23@tcp [ 1528.750783] LNetError: 68700:0:(api-ni.c:4395:lnet_ni_set_healthv()) manually adding local NI 192.168.204.23@tcp to recovery [ 1528.992550] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1528.993867] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1530.018234] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1531.155591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1534.113510] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1534.721878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1534.790464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1534.872275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.907603] LNet: 68797:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.919337] LNetError: 68797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.969092] LNet: Removed LNI 192.168.204.23@tcp [ 1535.980188] LNetError: 68797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.020472] LNet: Removed LNI 192.168.204.23@tcp1 [ 1536.069668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.134303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.169936] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1537.189820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1537.208244] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1538.231651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 1538.278619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1538.384984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1538.455457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1538.496606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1538.534962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1538.573283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1538.639350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.23@tcp [ 1538.675916] LNetError: 69070:0:(api-ni.c:4395:lnet_ni_set_healthv()) manually adding local NI 192.168.204.23@tcp to recovery [ 1539.168537] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1542.326669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1545.312368] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1545.321370] LNet: Skipped 1 previous similar message [ 1550.478993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1550.518681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1550.557463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1558.567477] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:27:05 (1780968425) [ 1558.695580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.736886] LNet: 69789:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.741539] LNetError: 69789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.748973] LNet: Removed LNI 192.168.204.23@tcp [ 1558.777255] LNet: Removed LNI 192.168.204.23@tcp1 [ 1558.830528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.960093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.014173] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1559.853020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1559.887092] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1561.093942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 1561.158535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1561.220685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1561.267905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1561.311789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1561.373356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1561.415377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.23@tcp --health 0 [ 1561.600712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1561.953351] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.23@tcp [ 1562.977793] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1562.999917] LNet: Skipped 1 previous similar message [ 1577.712166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1577.946625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1577.973655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.006791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.23@tcp --health 500 [ 1578.115480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1585.099409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1585.218530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.251452] LNet: 70287:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.262569] LNetError: 70287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.276360] LNetError: 70287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1585.283357] LNet: Removed LNI 192.168.204.23@tcp [ 1585.292510] LNet: Removed LNI 192.168.204.23@tcp1 [ 1585.323684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.390944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.416520] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1586.457294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1586.518892] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1587.818680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 1587.890875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1587.971638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1588.003916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1588.040118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1588.123778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1588.177910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1588.252204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.23@tcp --health 0 [ 1589.476611] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1589.494642] LNet: Skipped 3 previous similar messages [ 1589.661413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1600.205704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1600.273916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1600.309292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1608.147858] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:27:55 (1780968475) [ 1608.975656] LNet: 71343:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.985731] LNetError: 71343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.999942] LNetError: 71343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.021829] LNet: Removed LNI 192.168.204.23@tcp [ 1609.039575] LNet: Removed LNI 192.168.204.23@tcp1 [ 1609.782919] Key type .llcrypt unregistered [ 1609.787357] Key type ._llcrypt unregistered [ 1616.465734] Key type ._llcrypt registered [ 1616.470389] Key type .llcrypt registered [ 1616.661529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1616.676172] alg: No test for adler32 (adler32-zlib) [ 1617.649928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1617.693419] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1627.377577] Lustre: DEBUG MARKER: Initial discovery [ 1627.544227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.123@tcp [ 1630.646762] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1630.731674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1630.809332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.123@tcp [ 1630.851634] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1630.905978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1631.007558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.123@tcp -r 1 -m GET -l 3 [ 1631.009493] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1631.074955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.123@tcp -r 1 -m GET -e local_error [ 1631.187951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.123@tcp -r 1 -m PUT -l 6 [ 1633.158692] Lustre: DEBUG MARKER: Force oleg423-client.virtnet to discover 192.168.204.123@tcp (in background) [ 1633.263471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.123@tcp [ 1634.894374] Lustre: DEBUG MARKER: Force oleg423-server to discover 192.168.204.23@tcp [ 1636.320929] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1637.506287] Lustre: DEBUG MARKER: Wait for 72105 [ 1639.359511] Lustre: DEBUG MARKER: Finished wait on 72105 [ 1639.412354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1639.466689] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1639.485902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1639.888942] LNet: 72569:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.895490] LNetError: 72569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.904780] LNet: Removed LNI 192.168.204.23@tcp [ 1640.300986] Key type .llcrypt unregistered [ 1640.302560] Key type ._llcrypt unregistered [ 1656.052633] Key type ._llcrypt registered [ 1656.057853] Key type .llcrypt registered [ 1657.561797] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:28:44 (1780968524) [ 1658.059405] Key type .llcrypt unregistered [ 1658.065666] Key type ._llcrypt unregistered [ 1663.784188] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1664.014802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1664.020623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1664.024256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1666.547691] Key type ._llcrypt registered [ 1666.550434] Key type .llcrypt registered [ 1666.807023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.824508] alg: No test for adler32 (adler32-zlib) [ 1667.742912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.801657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.834545] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1668.451365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1668.470063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1676.452981] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:29:03 (1780968543) [ 1676.905910] LNet: 74667:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.913246] LNetError: 74667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.928441] LNet: Removed LNI 192.168.204.23@tcp [ 1676.948173] LNet: Removed LNI 10.1.2.3@tcp [ 1677.501502] Key type .llcrypt unregistered [ 1677.506674] Key type ._llcrypt unregistered [ 1683.555033] Key type ._llcrypt registered [ 1683.559899] Key type .llcrypt registered [ 1683.761668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1683.777365] alg: No test for adler32 (adler32-zlib) [ 1684.741653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.822573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.859730] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1685.626287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1685.643777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1686.457991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.23@tcp [ 1687.521585] LNetError: Unexpected error -22 connecting to 192.168.204.23@tcp at host 192.168.204.23:988 [ 1692.553233] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:29:19 (1780968559) [ 1692.715798] LNet: 75804:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.725401] LNetError: 75804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.732092] LNet: Removed LNI 192.168.204.23@tcp [ 1692.745289] LNet: Removed LNI 10.1.2.3@tcp [ 1693.326496] Key type .llcrypt unregistered [ 1693.329154] Key type ._llcrypt unregistered [ 1699.164148] Key type ._llcrypt registered [ 1699.170753] Key type .llcrypt registered [ 1699.412134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1699.432763] alg: No test for adler32 (adler32-zlib) [ 1700.456207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.519713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1700.565885] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1701.418657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1701.436487] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 1702.269963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.23@tcp1 --nid 192.168.204.23@tcp2 [ 1706.747287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp1 192.168.204.23@tcp2 [ 1706.782897] LNetError: 76646:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.23@tcp1 to 192.168.204.23@tcp2 - Route table may be misconfigured [ 1706.832364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp2 192.168.204.23@tcp1 [ 1713.242845] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:29:40 (1780968580) [ 1713.396849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.423716] LNet: 77225:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.441420] LNetError: 77225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.456821] LNet: Removed LNI 192.168.204.23@tcp1 [ 1714.536246] LNet: Removed LNI 192.168.204.23@tcp2 [ 1714.596683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.678951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.719288] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1715.433241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1715.467758] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1716.587744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 1716.643761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.23@tcp [ 1716.696667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp1 192.168.204.23@tcp1 [ 1716.750224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.23@tcp -d 192.168.204.23@tcp -e network_timeout [ 1716.793770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -e network_timeout [ 1716.854112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -e network_timeout [ 1716.905132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -e network_timeout [ 1716.941863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1716.965055] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1716.986914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1717.111460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1717.214919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1717.732179] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1724.081898] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:29:50 (1780968590) [ 1724.207981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.245392] LNet: 78091:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.252837] LNetError: 78091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.256331] LNetError: 78091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1724.274156] LNet: Removed LNI 192.168.204.23@tcp [ 1724.310689] LNet: Removed LNI 192.168.204.23@tcp1 [ 1724.344245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.462431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1725.130209] Key type .llcrypt unregistered [ 1725.134079] Key type ._llcrypt unregistered [ 1738.206380] Key type ._llcrypt registered [ 1738.209450] Key type .llcrypt registered [ 1739.786923] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:30:06 (1780968606) [ 1740.336884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.348649] alg: No test for adler32 (adler32-zlib) [ 1741.320706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.113885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1745.292420] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:30:12 (1780968612) [ 1745.762808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.791909] LNet: 79247:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.809863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.879534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.913335] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1746.708468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1746.743536] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1748.015691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1748.079906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp1 [ 1748.119381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp1 [ 1755.417086] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:30:22 (1780968622) [ 1758.101739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg423-server [ 1760.237720] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:30:26 (1780968626) [ 1762.332477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg423-server [ 1764.383874] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:30:31 (1780968631) [ 1766.723956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg423-server [ 1768.833686] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:30:35 (1780968635) [ 1770.975560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg423-server [ 1773.441575] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:30:39 (1780968639) [ 1775.902317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg423-server [ 1777.912088] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:30:44 (1780968644) [ 1780.789333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg423-server [ 1782.792364] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:30:49 (1780968649) [ 1785.224308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg423-server [ 1787.540424] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:30:54 (1780968654) [ 1790.680921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg423-server [ 1793.030392] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:30:59 (1780968659) [ 1796.165516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg423-server [ 1798.627379] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:31:05 (1780968665) [ 1798.820731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.850241] LNet: 83188:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.857645] LNetError: 83188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.904218] LNet: Removed LNI 192.168.204.23@tcp [ 1799.907507] LNetError: 83188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.953431] LNet: Removed LNI 192.168.204.23@tcp1 [ 1800.011228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.125531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1800.161343] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1801.026213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1801.127674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1801.440198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.491273] LNet: 83335:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.504975] LNetError: 83335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.533643] LNet: Removed LNI 192.168.204.23@tcp [ 1801.592765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.716525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1801.764476] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1803.032529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1803.362519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.461419] LNet: Removed LNI 192.168.204.23@tcp [ 1804.534324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.673748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.746313] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1805.903720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1806.111610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1806.396671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.417976] LNet: 83628:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.423880] LNet: 83628:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1806.433316] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.445080] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1806.459300] LNet: Removed LNI 192.168.204.23@tcp [ 1806.493158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.574891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1806.618513] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1807.923647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1808.202702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.259338] LNet: Removed LNI 192.168.204.23@tcp [ 1809.333051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.430796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.492465] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1810.546877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1810.654923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1811.013072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.060755] LNet: 83920:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.066856] LNet: 83920:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1811.091087] LNetError: 83920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.106964] LNetError: 83920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1811.161350] LNet: Removed LNI 192.168.204.23@tcp [ 1811.201905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.295673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1811.382951] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1812.671705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1813.158802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.268669] LNet: Removed LNI 192.168.204.23@tcp [ 1814.337880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.464778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1814.531714] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1815.763625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1815.876545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1817.323835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.420699] LNet: Removed LNI 192.168.204.23@tcp [ 1818.475385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.561312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1818.622098] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1819.861118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1820.254059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.304038] LNet: 84368:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.309805] LNet: 84368:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1820.320482] LNetError: 84368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.325733] LNetError: 84368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1820.351175] LNet: Removed LNI 192.168.204.23@tcp [ 1820.402834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.493837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.534322] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1821.460860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1821.556470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1823.238885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.314212] LNet: Removed LNI 192.168.204.23@tcp [ 1824.362552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.480608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1824.509668] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1825.705972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1826.094494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.176860] LNet: Removed LNI 192.168.204.23@tcp [ 1827.211660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.297664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.330477] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1828.341350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1828.429824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1828.872983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.930927] LNet: Removed LNI 192.168.204.23@tcp [ 1829.995404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.129590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1830.188900] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1831.292546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 1831.657610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.743534] LNet: Removed LNI 192.168.204.23@tcp [ 1832.820111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.952301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.985239] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1833.971170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1834.076932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.117412] LNet: Removed LNI 192.168.204.23@tcp [ 1834.156262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.208575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.228580] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1835.355827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1845.145038] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:31:51 (1780968711) [ 1845.650847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.688615] LNet: 85808:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.699854] LNet: 85808:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1845.721462] LNetError: 85808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.737417] LNetError: 85808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1845.768844] LNet: Removed LNI 192.168.204.23@tcp [ 1845.809467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.912559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1845.966678] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/180] [ 1847.834326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.883718] LNet: Removed LNI 192.168.204.23@tcp231 [ 1847.928414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.095492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1848.151054] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/180] [ 1848.156719] LNet: 85954:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1849.131656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1849.173108] LNet: Removed LNI 192.168.204.23@tcp231 [ 1849.218124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1849.245914] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/180] [ 1850.751432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.815463] LNet: Removed LNI 192.168.204.23@tcp231 [ 1850.853643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.936533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1850.983410] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/0] [ 1850.990438] LNet: 86107:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.687219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1851.755660] LNet: Removed LNI 192.168.204.23@tcp231 [ 1851.780578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1851.825705] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/0] [ 1853.445388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.488442] LNet: Removed LNI 192.168.204.23@tcp231 [ 1853.557907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.615302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1853.669618] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/60] [ 1853.674309] LNet: 86258:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1854.415570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1854.471715] LNet: Removed LNI 192.168.204.23@tcp231 [ 1854.506316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1854.536883] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/60] [ 1856.050852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.122665] LNet: Removed LNI 192.168.204.23@tcp231 [ 1856.197267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.304846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1856.342642] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/180] [ 1856.352240] LNet: 86410:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.077418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1857.132418] LNet: Removed LNI 192.168.204.23@tcp231 [ 1857.154286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1857.209801] LNet: Added LNI 192.168.204.23@tcp231 [8/256/0/180] [ 1866.037130] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:32:12 (1780968732) [ 1866.398860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.422383] LNet: Removed LNI 192.168.204.23@tcp231 [ 1866.439841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.482684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1866.503628] LNet: Added LNI 192.168.204.23@tcp232 [8/256/0/180] [ 1867.413699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1867.484869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1867.536533] LNet: Removed LNI 192.168.204.23@tcp232 [ 1867.569809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1867.619166] LNet: Added LNI 192.168.204.23@tcp232 [8/256/0/180] [ 1868.439960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1868.484468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1868.537083] LNet: Removed LNI 192.168.204.23@tcp232 [ 1868.564831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1868.596237] LNet: Added LNI 192.168.204.23@tcp232 [8/256/0/180] [ 1869.254497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1869.286333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1869.319987] LNet: Removed LNI 192.168.204.23@tcp232 [ 1869.352631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1869.388698] LNet: Added LNI 192.168.204.23@tcp232 [8/256/0/180] [ 1870.237638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1870.288761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1870.321878] LNet: Removed LNI 192.168.204.23@tcp232 [ 1870.355460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1877.249739] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:32:23 (1780968743) [ 1877.925384] LNet: 88268:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.930363] LNet: 88268:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1878.709399] Key type .llcrypt unregistered [ 1878.711894] Key type ._llcrypt unregistered [ 1884.262587] Key type ._llcrypt registered [ 1884.264995] Key type .llcrypt registered [ 1884.519745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.545960] alg: No test for adler32 (adler32-zlib) [ 1885.698818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1885.734509] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1894.035631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 1903.944933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.978705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1904.001757] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1906.040461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_interrupt -r 2 [ 1906.113734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_interrupt -r 2 [ 1906.170606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_interrupt -r 2 [ 1906.211628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_interrupt -r 2 [ 1906.274111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_interrupt -r 2 [ 1906.322943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_interrupt -r 2 [ 1906.379808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_interrupt -r 2 [ 1906.443684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_interrupt -r 2 [ 1906.514897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1906.554603] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1906.567704] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1906.860490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1906.909502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1906.967878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1907.006944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1907.048825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1907.099415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1907.152550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.192555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1907.212600] LNet: 88932:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1907.244123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1907.281894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1907.301817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1907.318444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1907.369553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1907.418792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1908.100988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_dropped -r 2 [ 1908.163655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_dropped -r 2 [ 1908.247225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_dropped -r 2 [ 1908.355437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_dropped -r 2 [ 1908.426339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_dropped -r 2 [ 1908.464402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_dropped -r 2 [ 1908.497757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_dropped -r 2 [ 1908.549280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_dropped -r 2 [ 1908.610270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1908.639788] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1908.887291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1908.914102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1908.945692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1908.963960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1908.993308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1909.031487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1909.059097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1909.121212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1909.151607] LNet: 89052:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1909.181995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1909.209395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1909.231851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1909.262923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1909.293914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1909.329264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1909.745654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_aborted -r 2 [ 1909.782734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_aborted -r 2 [ 1909.812365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_aborted -r 2 [ 1909.870818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_aborted -r 2 [ 1909.917085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_aborted -r 2 [ 1909.993788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_aborted -r 2 [ 1910.035855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_aborted -r 2 [ 1910.076972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_aborted -r 2 [ 1910.141870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1910.162751] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1910.378580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1910.412625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1910.440473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1910.466658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1910.488283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1910.519040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1910.544699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1910.587904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1910.603778] LNet: 89172:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1910.617866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1910.651627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1910.707919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1910.758095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1910.806754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1910.856869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1911.414182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_no_route -r 2 [ 1911.445352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_no_route -r 2 [ 1911.473742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_no_route -r 2 [ 1911.503606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_no_route -r 2 [ 1911.535590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_no_route -r 2 [ 1911.571375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_no_route -r 2 [ 1911.614687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_no_route -r 2 [ 1911.654090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_no_route -r 2 [ 1911.682863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1912.005463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1912.071911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1912.113989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1912.177787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1912.200706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1912.222742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1912.263737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1912.317133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1912.384392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1912.464487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1912.505257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1912.547197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1912.581863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1912.614601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1913.236226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e local_timeout -r 2 [ 1913.309298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e local_timeout -r 2 [ 1913.374852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e local_timeout -r 2 [ 1913.424352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e local_timeout -r 2 [ 1913.507660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e local_timeout -r 2 [ 1913.583949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e local_timeout -r 2 [ 1913.620845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e local_timeout -r 2 [ 1913.671068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e local_timeout -r 2 [ 1913.765468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1913.791389] LNet: There was an unexpected network error while writing to 192.168.204.123: rc = -22 [ 1913.796521] LNet: Skipped 1 previous similar message [ 1914.096244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1914.134918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1914.187548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1914.237506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1914.278717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1914.309979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1914.367029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1914.412929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1914.481754] LNet: 89413:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1914.489094] LNet: 89413:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1914.503237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1914.554195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1914.604783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1914.676142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1914.719709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1914.779394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1915.335894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp -m GET -e remote_dropped -r 2 [ 1915.393473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.123@tcp1 -m GET -e remote_dropped -r 2 [ 1915.464573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp -m GET -e remote_dropped -r 2 [ 1915.511903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp -d 192.168.204.23@tcp1 -m GET -e remote_dropped -r 2 [ 1915.551716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp -m GET -e remote_dropped -r 2 [ 1915.612239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.123@tcp1 -m GET -e remote_dropped -r 2 [ 1915.656629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp -m GET -e remote_dropped -r 2 [ 1915.718657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.23@tcp1 -d 192.168.204.23@tcp1 -m GET -e remote_dropped -r 2 [ 1915.781483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.23@tcp 192.168.204.123@tcp [ 1915.812695] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.123@tcp [ 1916.098630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1916.129757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1916.151714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1916.178239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1916.213684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1916.253795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1916.287945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1916.320226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1916.360376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1916.397331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1916.431466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1916.468955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1916.494762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1916.532878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.032402] LNet: 89607:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.042938] LNetError: 89607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.055043] LNet: Removed LNI 192.168.204.23@tcp [ 1922.070525] LNet: Removed LNI 192.168.204.23@tcp1 [ 1922.472616] Key type .llcrypt unregistered [ 1922.477874] Key type ._llcrypt unregistered [ 1937.899382] Key type ._llcrypt registered [ 1937.901196] Key type .llcrypt registered [ 1939.261796] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:33:26 (1780968806) [ 1939.670260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.681712] alg: No test for adler32 (adler32-zlib) [ 1940.624522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.779922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1940.840361] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1941.882799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1941.913885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.23@tcp [ 1941.939682] LNetError: 90613:0:(api-ni.c:4395:lnet_ni_set_healthv()) manually adding local NI 192.168.204.23@tcp to recovery [ 1942.077093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.23@tcp [ 1942.240595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1949.716302] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:33:36 (1780968816) [ 1949.837039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.890830] LNet: 91205:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.900052] LNetError: 91205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.922930] LNet: Removed LNI 192.168.204.23@tcp [ 1949.958665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.080374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.108419] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1951.045663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1951.065145] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 1952.258552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.23@tcp [ 1952.287133] LNetError: 91477:0:(api-ni.c:4395:lnet_ni_set_healthv()) manually adding local NI 192.168.204.23@tcp to recovery [ 1953.120886] LNet: 1 local NIs in recovery (showing 1): 192.168.204.23@tcp [ 1953.123039] LNet: There was an unexpected network error while writing to 192.168.204.23: rc = -22 [ 1953.584097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.23@tcp [ 1962.396748] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:33:49 (1780968829) [ 1964.626594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg423-server [ 1966.298281] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:33:53 (1780968833) [ 1968.283439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg423-server [ 1970.045064] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:33:57 (1780968837) [ 1971.755194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg423-server [ 1973.906049] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:34:00 (1780968840) [ 1973.991895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.023236] LNet: 93128:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.032051] LNetError: 93128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.040428] LNet: Removed LNI 192.168.204.23@tcp [ 1974.074326] LNet: Removed LNI 192.168.204.23@tcp1 [ 1974.106064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.187754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1974.213549] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1974.783463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1974.833752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1974.895438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1974.968605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1975.012457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1975.056187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1981.292723] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:34:08 (1780968848) [ 1981.570374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.592560] LNet: 93868:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.598868] LNetError: 93868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.603878] LNetError: 93868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1981.617358] LNet: Removed LNI 192.168.204.23@tcp [ 1981.632104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.683602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.717370] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1989.041319] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:34:15 (1780968855) [ 1989.130747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.161232] LNet: 94571:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.167580] LNetError: 94571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.192723] LNet: Removed LNI 192.168.204.23@tcp [ 1989.240073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.320850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.345871] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 1996.075262] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:34:23 (1780968863) [ 1997.783594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1999.636242] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:34:26 (1780968866) [ 2000.467437] LNet: 95628:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.474309] LNetError: 95628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.488199] LNet: Removed LNI 192.168.204.23@tcp [ 2001.194166] Key type .llcrypt unregistered [ 2001.209053] Key type ._llcrypt unregistered [ 2008.654350] Key type ._llcrypt registered [ 2008.657751] Key type .llcrypt registered [ 2008.933947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.962284] alg: No test for adler32 (adler32-zlib) [ 2009.994509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2010.038398] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2018.544654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 2023.039792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.123@tcp [ 2023.406405] LNet: 96048:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.421715] LNetError: 96048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.442669] LNet: Removed LNI 192.168.204.23@tcp [ 2023.872531] Key type .llcrypt unregistered [ 2023.876320] Key type ._llcrypt unregistered [ 2036.582943] Key type ._llcrypt registered [ 2036.587914] Key type .llcrypt registered [ 2038.055452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2040.163263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2042.243454] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:35:09 (1780968909) [ 2042.762594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.802929] alg: No test for adler32 (adler32-zlib) [ 2043.720394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.817631] LNet: 97149:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.361489] Key type .llcrypt unregistered [ 2044.363769] Key type ._llcrypt unregistered [ 2050.523899] Key type ._llcrypt registered [ 2050.537604] Key type .llcrypt registered [ 2050.717656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.756229] alg: No test for adler32 (adler32-zlib) [ 2051.680600] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2051.683139] LNet: 97407:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2051.688764] LNet: 97407:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2051.724796] LNetError: 97419:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 2051.739529] LNetError: 97419:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) route to o2ib through 192.168.204.24@tcp has gone from up to down [ 2052.243712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.265383] LNet: 97465:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.272307] LNetError: 97465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.292239] LNet: Removed LNI 192.168.204.23@tcp [ 2052.310263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.364134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2052.399862] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2052.404548] LNet: 97472:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2052.410304] LNet: 97472:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 2052.425383] LNetError: 97472:0:(peer.c:367:lnet_peer_ni_del_locked()) Peer NI 192.168.204.24@tcp is a gateway. Can not delete it [ 2052.442276] LNetError: 97472:0:(peer.c:2167:lnet_del_peer_ni()) 192.168.204.24@tcp is a router. Can not be deleted [ 2052.461430] LNet: 97472:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2052.465487] LNetError: 97469:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.27@tcp is being used as a gateway but routing feature is not turned on [ 2052.505850] LNetError: 97469:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2052.521713] LNetError: 97469:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) route to o2ib through 192.168.204.27@tcp has gone from up to down [ 2052.532442] LNetError: 97469:0:(lib-lnet.h:1306:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2059.457383] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:35:25 (1780968925) [ 2061.895562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg423-server [ 2063.599976] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:35:30 (1780968930) [ 2065.879724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg423-server [ 2068.109565] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:35:34 (1780968934) [ 2070.006789] Lustre: Lustre: Build Version: 2.17.52_126_g9f56455 [ 2071.944952] Key type lgssc registered [ 2073.114835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.387715] Key type lgssc unregistered [ 2093.838582] LNet: 99952:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.870047] LNetError: 99952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.896268] LNet: Removed LNI 192.168.204.23@tcp [ 2094.518570] Key type .llcrypt unregistered [ 2094.522683] Key type ._llcrypt unregistered [ 2108.766477] Key type ._llcrypt registered [ 2108.769243] Key type .llcrypt registered [ 2110.089109] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:36:16 (1780968976) [ 2110.711736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.736187] alg: No test for adler32 (adler32-zlib) [ 2111.793979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.910451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2111.997739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2112.090659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2112.149357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2120.063135] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:36:26 (1780968986) [ 2120.194284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.230315] LNet: 101369:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.278426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.374982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.420893] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2121.601370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.631371] LNet: 101520:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.643976] LNetError: 101520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.664251] LNet: Removed LNI 192.168.204.23@tcp [ 2121.708433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.779385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2121.811524] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2122.341438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.356945] LNetError: 101548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.372603] LNet: Removed LNI 192.168.204.23@tcp [ 2122.405772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.477837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2122.524906] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2129.594230] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:36:36 (1780968996) [ 2129.662672] LNet: 102139:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.667145] LNet: 102139:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2129.676513] LNetError: 102139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.697853] LNet: Removed LNI 192.168.204.23@tcp [ 2130.468780] Key type .llcrypt unregistered [ 2130.475678] Key type ._llcrypt unregistered [ 2137.814583] Key type ._llcrypt registered [ 2137.816526] Key type .llcrypt registered [ 2138.045523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2138.059969] alg: No test for adler32 (adler32-zlib) [ 2139.059617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.193343] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2139.195719] LNet: *** cfs_fail_loc=e003, val=100*** [ 2140.056240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 2140.083675] LNet: *** cfs_fail_loc=e003, val=100*** [ 2140.088117] LNet: Skipped 2 previous similar messages [ 2140.095759] Ping reply received with mapping: [ 2140.098340] found metadata reply NI[192.168.204.23@tcp] [ 2140.104502] 17 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2140.107336] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2140.110526] 5a 5a 5a 5a 5a 5a 5a 5a [ 2140.191808] LNet: 102548:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.195809] LNetError: 102548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.210544] LNet: Removed LNI 192.168.204.23@tcp [ 2140.740480] Key type .llcrypt unregistered [ 2140.749092] Key type ._llcrypt unregistered [ 2153.414814] Key type ._llcrypt registered [ 2153.418334] Key type .llcrypt registered [ 2154.767299] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:37:01 (1780969021) [ 2155.298417] Key type .llcrypt unregistered [ 2155.304267] Key type ._llcrypt unregistered [ 2160.927646] Key type ._llcrypt registered [ 2160.933802] Key type .llcrypt registered [ 2161.145242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2161.168660] alg: No test for adler32 (adler32-zlib) [ 2162.337923] LNetError: 103605:0:(api-ni.c:2767:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2162.832822] Key type .llcrypt unregistered [ 2162.839069] Key type ._llcrypt unregistered [ 2168.415863] Key type ._llcrypt registered [ 2168.418936] Key type .llcrypt registered [ 2169.791740] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:37:16 (1780969036) [ 2170.320540] Key type .llcrypt unregistered [ 2170.326147] Key type ._llcrypt unregistered [ 2176.027638] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2176.031381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2178.742266] Key type ._llcrypt registered [ 2178.747340] Key type .llcrypt registered [ 2178.946895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2178.959966] alg: No test for adler32 (adler32-zlib) [ 2180.045690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.173255] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2191.281479] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:37:38 (1780969058) [ 2191.366982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.396436] LNet: 105394:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.405317] LNetError: 105394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2191.443754] LNet: Removed LNI 10.1.2.3@tcp [ 2191.494470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.464993] Key type .llcrypt unregistered [ 2192.468307] Key type ._llcrypt unregistered [ 2198.776828] Key type ._llcrypt registered [ 2198.779281] Key type .llcrypt registered [ 2199.020918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2199.043599] alg: No test for adler32 (adler32-zlib) [ 2199.955269] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2200.035445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 2200.239885] LNet: 105693:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.252198] LNetError: 105693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.272803] LNet: Removed LNI 192.168.204.23@tcp [ 2200.911667] Key type .llcrypt unregistered [ 2200.914710] Key type ._llcrypt unregistered [ 2212.272441] Key type ._llcrypt registered [ 2212.274406] Key type .llcrypt registered [ 2213.700049] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:38:00 (1780969080) [ 2214.552085] Key type .llcrypt unregistered [ 2214.564920] Key type ._llcrypt unregistered [ 2221.887613] Key type ._llcrypt registered [ 2221.891212] Key type .llcrypt registered [ 2222.260963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2222.287417] alg: No test for adler32 (adler32-zlib) [ 2231.051542] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:38:17 (1780969097) [ 2231.336529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.403612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.458856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2231.502636] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2232.266223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.309603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2232.337476] LNet: Removed LNI 192.168.204.23@tcp [ 2232.528627] LNet: 107457:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.968502] Key type .llcrypt unregistered [ 2232.970275] Key type ._llcrypt unregistered [ 2243.291464] Key type ._llcrypt registered [ 2243.293309] Key type .llcrypt registered [ 2244.484934] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:38:31 (1780969111) [ 2244.909950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.936391] alg: No test for adler32 (adler32-zlib) [ 2245.863989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.916427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.935573] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2246.602808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 2246.633481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.23@tcp [ 2246.648654] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2246.651219] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2246.654193] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2246, 0, 0 [ 2246.658937] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2246.661504] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2246.664371] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2246.666849] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2246, 0, 0 [ 2246.669781] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2246.671976] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2246.674340] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2246.676342] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2246, 0, 2296 [ 2246.679666] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2246.681070] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2246.683744] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2246.686555] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2246, 0, 2296 [ 2246.690032] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2252.746790] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:38:39 (1780969119) [ 2253.253467] LNet: 109017:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.262715] LNetError: 109017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.282393] LNet: Removed LNI 192.168.204.23@tcp [ 2253.948474] Key type .llcrypt unregistered [ 2253.952807] Key type ._llcrypt unregistered [ 2259.427962] Key type ._llcrypt registered [ 2259.430317] Key type .llcrypt registered [ 2259.725319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2259.757587] alg: No test for adler32 (adler32-zlib) [ 2260.666408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2260.703827] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2266.911701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 2274.783304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.852423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2274.883158] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 2275.845801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2275.967912] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2275.975747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2275.984908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2278.380110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2278.406085] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2287.739173] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:39:14 (1780969154) [ 2288.149343] LNet: 110534:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.164363] LNetError: 110534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.176499] LNet: Removed LNI 192.168.204.23@tcp [ 2289.185299] LNet: Removed LNI 192.168.204.23@tcp1 [ 2289.187492] LNetError: 110534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2289.207961] LNetError: 110534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2289.230643] LNet: Removed LNI 10.1.2.3@tcp99 [ 2290.151188] Key type .llcrypt unregistered [ 2290.155534] Key type ._llcrypt unregistered [ 2295.667601] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2295.671572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2298.180472] Key type ._llcrypt registered [ 2298.181535] Key type .llcrypt registered [ 2298.316812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2298.330778] alg: No test for adler32 (adler32-zlib) [ 2299.218508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.264810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.297537] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2300.047071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2300.075965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2302.261982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2302.328792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2302.369421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2302.394957] LNetError: 111280:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2302.427194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2302.460945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2302.959585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2303.035257] LNet: 111300:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.043548] LNetError: 111300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.052775] LNet: Removed LNI 192.168.204.23@tcp [ 2303.075729] LNet: Removed LNI 10.1.2.3@tcp [ 2303.752867] Key type .llcrypt unregistered [ 2303.755096] Key type ._llcrypt unregistered [ 2317.165245] Key type ._llcrypt registered [ 2317.170986] Key type .llcrypt registered [ 2318.420521] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:39:45 (1780969185) [ 2318.838855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.861978] alg: No test for adler32 (adler32-zlib) [ 2319.803871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.861389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.882842] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2320.748682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.23@tcp [ 2320.817547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg423-client@tcp [ 2326.615156] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:39:53 (1780969193) [ 2326.888816] LNet: 112852:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.901250] LNetError: 112852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.927720] LNet: Removed LNI 192.168.204.23@tcp [ 2327.598143] Key type .llcrypt unregistered [ 2327.602188] Key type ._llcrypt unregistered [ 2333.808417] Key type ._llcrypt registered [ 2333.814215] Key type .llcrypt registered [ 2333.981450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.002090] alg: No test for adler32 (adler32-zlib) [ 2334.963334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2335.008061] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2335.064212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2341.180888] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:40:08 (1780969208) [ 2342.423275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2344.341652] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:40:11 (1780969211) [ 2345.814558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2347.473506] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:40:14 (1780969214) [ 2347.572430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.611933] LNet: 114283:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.626077] LNetError: 114283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.643822] LNet: Removed LNI 192.168.204.23@tcp [ 2347.663649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.745804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.764479] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2348.605314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2348.653265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2355.822419] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:40:22 (1780969222) [ 2355.941106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.984880] LNet: 115000:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.996071] LNetError: 115000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.033500] LNet: Removed LNI 192.168.204.23@tcp [ 2356.059162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.117749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2356.154835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2356.897809] Key type .llcrypt unregistered [ 2356.900819] Key type ._llcrypt unregistered [ 2366.766970] Key type ._llcrypt registered [ 2366.770177] Key type .llcrypt registered [ 2368.125878] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:40:34 (1780969234) [ 2368.594132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.619975] alg: No test for adler32 (adler32-zlib) [ 2369.551732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.605514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.632617] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2370.308216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2370.373494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2377.277891] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:40:43 (1780969243) [ 2377.480145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.494974] LNet: 116579:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.499773] LNetError: 116579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.535766] LNet: Removed LNI 192.168.204.23@tcp [ 2378.565138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.656527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2378.690750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2385.731670] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:40:52 (1780969252) [ 2385.963693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2385.984820] LNet: 117167:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.007563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.080698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2386.606647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2386.644136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2386.718898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2387.311507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2387.350392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2387.449082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.473912] LNet: 117200:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.502375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.580511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2388.110812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2388.186955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2388.249494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2388.885679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2388.953850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2395.582349] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:41:02 (1780969262) [ 2395.789512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.808926] LNet: 117809:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.831538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.889801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2395.947407] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2396.559676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2396.599721] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2397.542230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2397.618058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2397.849595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.881427] LNetError: 118088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.895055] LNet: Removed LNI 192.168.204.23@tcp [ 2397.909159] LNet: Removed LNI 192.168.204.23@tcp2 [ 2397.946311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.049783] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2398.055466] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2398.059814] LNet: 118094:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2398.627376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2398.874474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.900365] LNetError: 118131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.914852] LNetError: 118131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2398.931047] LNet: Removed LNI 192.168.204.23@tcp [ 2398.947888] LNet: Removed LNI 192.168.204.23@tcp2 [ 2398.969962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.048666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.102300] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2399.778116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2399.802866] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2400.671157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2400.725881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2400.775129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2401.006470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.035942] LNet: 118416:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.049718] LNet: 118416:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2401.057925] LNetError: 118416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.066736] LNetError: 118416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2401.081237] LNet: Removed LNI 192.168.204.23@tcp [ 2402.151822] LNet: Removed LNI 192.168.204.23@tcp2 [ 2402.182606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.235973] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2402.240359] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2402.248144] LNet: 118422:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2402.656735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2402.838991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.857205] LNet: Removed LNI 192.168.204.23@tcp [ 2403.879062] LNet: Removed LNI 192.168.204.23@tcp2 [ 2403.903690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.965418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.003230] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2404.730618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2404.750034] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2405.735061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2405.847537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2406.108186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.135659] LNetError: 118738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.142160] LNetError: 118738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2406.148582] LNet: Removed LNI 192.168.204.23@tcp [ 2406.161709] LNet: Removed LNI 192.168.204.23@tcp2 [ 2406.185561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.245729] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2406.258378] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2406.264451] LNet: 118744:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2406.914696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2407.137113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.160180] LNet: Removed LNI 192.168.204.23@tcp [ 2408.172439] LNet: Removed LNI 192.168.204.23@tcp2 [ 2408.215282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.284905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2408.316578] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2408.882915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2408.910545] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2409.780206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2409.821851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2409.889945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2410.151275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.196775] LNet: 119066:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.204598] LNet: 119066:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2411.296551] LNet: Removed LNI 192.168.204.23@tcp [ 2412.328148] LNet: Removed LNI 192.168.204.23@tcp2 [ 2412.370593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.451275] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2412.455601] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2412.460898] LNet: 119072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2413.013394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2413.189349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.213381] LNet: Removed LNI 192.168.204.23@tcp [ 2414.247181] LNet: Removed LNI 192.168.204.23@tcp2 [ 2414.297894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.366622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2414.389866] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2415.027722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2415.043166] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2415.707756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2415.742360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2415.780758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2416.092782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2416.111948] LNetError: 119406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.116883] LNetError: 119406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2416.120796] LNet: Removed LNI 192.168.204.23@tcp [ 2416.131621] LNet: Removed LNI 192.168.204.23@tcp2 [ 2416.145655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.188586] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2416.196893] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2416.540296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2416.822990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.888245] LNet: Removed LNI 192.168.204.23@tcp [ 2417.902200] LNet: Removed LNI 192.168.204.23@tcp2 [ 2417.927602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.984666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2418.006324] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2418.544426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2418.563752] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2419.280653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2419.309819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2419.335213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2419.387645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2419.683683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.706701] LNet: Removed LNI 192.168.204.23@tcp [ 2421.741074] LNet: Removed LNI 192.168.204.23@tcp2 [ 2421.783299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.888526] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2421.900969] LNet: Added LNI 192.168.204.23@tcp2 [8/256/0/180] [ 2421.906732] LNet: 119772:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.911543] LNet: 119772:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2422.338942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2428.221297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2429.665703] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:41:36 (1780969296) [ 2429.873602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.889772] LNet: 120541:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.895527] LNet: 120541:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2429.900585] LNet: Removed LNI 192.168.204.23@tcp [ 2430.957949] LNet: Removed LNI 192.168.204.23@tcp2 [ 2431.006840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.272596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.296281] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2431.832337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2431.851347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2432.632725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.654343] LNetError: 120814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.663755] LNetError: 120814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2432.672114] LNet: Removed LNI 192.168.204.23@tcp [ 2432.694770] LNet: Removed LNI 10.1.2.3@tcp [ 2432.726118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2432.798985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2432.832908] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2432.842059] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2433.305828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2433.352288] LNet: Removed LNI 192.168.204.23@tcp [ 2433.363414] LNet: Removed LNI 10.1.2.3@tcp [ 2440.331892] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:41:47 (1780969307) [ 2440.530363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.596127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.811770] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2441.038594] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2441.059976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2441.062976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2443.516146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.543726] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2444.552438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2444.575263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2446.833759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.23@tcp [ 2446.874346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2446.926903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.23@tcp -m PUT -e local_timeout -r 1 [ 2446.993937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2447.034923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2447.042392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2447.231988] LNetError: 121490:0:(lib-move.c:4971:lnet_parse()) 192.168.204.23@tcp, src 192.168.204.23@tcp: Dropping PUT (error -108 looking up sender) [ 2449.120364] LNet: Removed LNI 192.168.204.23@tcp [ 2449.130931] LNet: Removed LNI 10.1.2.3@tcp [ 2449.479557] Key type .llcrypt unregistered [ 2449.485671] Key type ._llcrypt unregistered [ 2460.100513] Key type ._llcrypt registered [ 2460.103482] Key type .llcrypt registered [ 2461.385417] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:42:08 (1780969328) [ 2461.701287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.717750] alg: No test for adler32 (adler32-zlib) [ 2462.610857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.922725] LNet: 122812:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.409497] Key type .llcrypt unregistered [ 2463.412292] Key type ._llcrypt unregistered [ 2470.411460] Key type ._llcrypt registered [ 2470.413815] Key type .llcrypt registered [ 2470.786820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2470.843030] alg: No test for adler32 (adler32-zlib) [ 2471.726373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2471.750996] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2478.934765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 2479.186784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2479.849827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.123@tcp [ 2479.969135] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.123@tcp [ 2480.136447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2480.161707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2480.190212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2480.215735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2480.235944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2480.259180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2480.291375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2480.327927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2480.354962] LNet: 123263:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2480.372126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.398136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.441361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.482487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.522877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.561390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2481.242877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2481.477247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.536725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2481.579141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.619666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.666748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.694343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.767298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.838898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.883608] LNet: 123381:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.916707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.942257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.966384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.996453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2482.031589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2482.070549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.722475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.123@tcp [ 2482.929243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2482.957442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2482.987568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.019825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.052943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.085665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.109587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.144826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2483.172155] LNet: 123498:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2483.200509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.243473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.279666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2483.320309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2483.367526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2483.403884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2483.907800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2484.126814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.155203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2484.182937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2484.216714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2484.239275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2484.274722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2484.318906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2484.360317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.407671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.450285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.479164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.503155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.544842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.580879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.820459] LNet: 123661:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.829688] LNetError: 123661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.844964] LNet: Removed LNI 192.168.204.23@tcp [ 2485.159941] Key type .llcrypt unregistered [ 2485.166087] Key type ._llcrypt unregistered [ 2500.660800] Key type ._llcrypt registered [ 2500.664574] Key type .llcrypt registered [ 2502.236735] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:42:48 (1780969368) [ 2502.930736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.957498] alg: No test for adler32 (adler32-zlib) [ 2504.024937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.607701] LNet: 124560:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.586431] Key type .llcrypt unregistered [ 2505.593688] Key type ._llcrypt unregistered [ 2513.015732] Key type ._llcrypt registered [ 2513.023899] Key type .llcrypt registered [ 2513.445263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2513.468722] alg: No test for adler32 (adler32-zlib) [ 2514.492346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2514.551530] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2522.003660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.123@tcp [ 2530.270378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.306442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2530.333863] LNet: Added LNI 192.168.204.23@tcp1 [8/256/0/180] [ 2531.452885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2532.050773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.123@tcp [ 2532.285577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.322732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2532.348815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2532.382895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2532.404835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2532.460772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2532.513262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2532.558887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2532.577353] LNet: 125214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2532.602770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2532.648631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2532.687532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2532.726264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2532.782916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2532.840962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2533.660748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2533.968865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2533.995750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2534.026152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.055207] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.123@tcp, 192.168.204.123@tcp1 [ 2534.070728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.113431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.161520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.224969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.261505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.273846] LNet: 125331:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.289369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.314082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.332807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.374981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.430569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.507680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.237835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.123@tcp [ 2535.453490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2535.488753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2535.528670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2535.560023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2535.591694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2535.617590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2535.647260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2535.673778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2535.687823] LNet: 125448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2535.702412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2535.750456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2535.786812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2535.841942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2535.894182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2535.937319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2536.535739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2536.743986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2536.801691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2536.882255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2536.923797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2536.966031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2537.040098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2537.075156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2537.101809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2537.150484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2537.187491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2537.220640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2537.259284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2537.305419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2537.343817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2541.875133] LNet: 125647:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2541.887268] LNetError: 125647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2541.897390] LNet: Removed LNI 192.168.204.23@tcp [ 2541.935611] LNet: Removed LNI 192.168.204.23@tcp1 [ 2542.434851] Key type .llcrypt unregistered [ 2542.439394] Key type ._llcrypt unregistered [ 2557.770725] Key type ._llcrypt registered [ 2557.774319] Key type .llcrypt registered [ 2559.175146] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:43:45 (1780969425) [ 2567.512851] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:43:54 (1780969434) [ 2575.790977] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:44:02 (1780969442) [ 2583.291055] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:44:10 (1780969450) [ 2590.396971] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:44:17 (1780969457) [ 2598.256996] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:44:25 (1780969465) [ 2605.112237] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:44:32 (1780969472) [ 2611.619206] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:44:38 (1780969478) [ 2618.604741] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:44:45 (1780969485) [ 2626.115258] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:44:52 (1780969492) [ 2632.742225] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:44:59 (1780969499) [ 2638.771650] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:45:05 (1780969505) [ 2646.004720] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:45:13 (1780969513) [ 2652.413337] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:45:19 (1780969519) [ 2660.669254] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:45:27 (1780969527) [ 2667.841589] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:45:34 (1780969534) [ 2675.394459] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:45:42 (1780969542) [ 2682.379184] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:45:49 (1780969549) [ 2682.770274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2682.790606] alg: No test for adler32 (adler32-zlib) [ 2683.788139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2683.923666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2683.965874] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2685.197845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2685.225566] LNet: 136503:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2685.236498] LNetError: 136503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.256822] LNet: Removed LNI 192.168.204.23@tcp [ 2685.279886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2685.336212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2685.359119] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2692.476931] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:45:59 (1780969559) [ 2700.745823] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:46:07 (1780969567) [ 2709.746433] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:46:16 (1780969576) [ 2718.287881] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:46:24 (1780969584) [ 2725.051170] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:46:31 (1780969591) [ 2733.239424] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:46:40 (1780969600) [ 2740.524939] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:46:47 (1780969607) [ 2747.805640] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:46:54 (1780969614) [ 2756.436957] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:47:03 (1780969623) [ 2765.500405] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:47:12 (1780969632) [ 2773.905253] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:47:20 (1780969640) [ 2780.597858] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:47:27 (1780969647) [ 2786.576439] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2525 sec ========== 21:47:33 (1780969653) [ 2786.901674] LNet: 144077:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.906505] LNetError: 144077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.916352] LNet: Removed LNI 192.168.204.23@tcp [ 2787.502474] Key type .llcrypt unregistered [ 2787.507150] Key type ._llcrypt unregistered [ 2800.176366] Key type ._llcrypt registered [ 2800.186229] Key type .llcrypt registered [ 2800.446460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2800.457798] alg: No test for adler32 (adler32-zlib) [ 2801.552220] Lustre: Lustre: Build Version: 2.17.52_126_g9f56455 [ 2801.844884] LNet: Added LNI 192.168.204.23@tcp [8/256/0/180] [ 2803.536842] Key type lgssc registered [ 2804.930796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.731461] Lustre: Mounted lustre-client [ 2898.480163] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2906.899119] LustreError: 146903:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8d7dc68ec800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2907.026705] Lustre: Unmounted lustre-client [ 2959.705456] Key type lgssc unregistered [ 2960.093480] LNet: 147588:0:(lib-ptl.c:980:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.103963] LNetError: 147588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.139444] LNet: Removed LNI 192.168.204.23@tcp [ 2960.669174] Key type .llcrypt unregistered [ 2960.677091] Key type ._llcrypt unregistered