[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509781783 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003314] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007872] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021529] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026108] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027122] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030363] *** VALIDATE proc *** [ 0.031209] *** VALIDATE cgroup *** [ 0.032000] *** VALIDATE cgroup2 *** [ 0.032000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033055] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035027] Spectre V2 : User space: Vulnerable [ 0.036011] Speculative Store Bypass: Vulnerable [ 0.039743] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.041888] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050310] rcu: Hierarchical SRCU implementation. [ 0.052490] smp: Bringing up secondary CPUs ... [ 0.053575] x86: Booting SMP configuration: [ 0.054046] .... node #0, CPUs: #1 #2 #3 [ 0.060520] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150043] node 0 deferred pages initialised in 85ms [ 0.154009] devtmpfs: initialized [ 0.155320] x86/mm: Memory block size: 128MB [ 0.159060] gcov: version magic: 0x41383552 [ 0.161303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164362] pinctrl core: initialized pinctrl subsystem [ 0.167179] [ 0.168008] ************************************************************* [ 0.170012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173016] ** ** [ 0.176014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178009] ** ** [ 0.181013] ** This means that this kernel is built to expose internal ** [ 0.183012] ** IOMMU data structures, which may compromise security on ** [ 0.186012] ** your system. ** [ 0.188010] ** ** [ 0.191014] ** If you see this message and you are not debugging the ** [ 0.194021] ** kernel, report this immediately to your vendor! ** [ 0.196010] ** ** [ 0.198011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199000] ************************************************************* [ 0.199891] NET: Registered protocol family 16 [ 0.201685] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213457] cpuidle: using governor menu [ 0.215805] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218489] PCI: Using configuration type 1 for base access [ 0.220130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229231] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231074] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235233] cryptd: max_cpu_qlen set to 1000 [ 0.238189] ACPI: Added _OSI(Module Device) [ 0.239013] ACPI: Added _OSI(Processor Device) [ 0.240010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246717] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252590] ACPI: Interpreter enabled [ 0.256098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257022] ACPI: Using IOAPIC for interrupt routing [ 0.260179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264660] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274824] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292165] acpiphp: Slot [2] registered [ 0.293107] acpiphp: Slot [5] registered [ 0.295135] acpiphp: Slot [6] registered [ 0.296150] acpiphp: Slot [3] registered [ 0.297092] acpiphp: Slot [4] registered [ 0.299099] acpiphp: Slot [7] registered [ 0.301112] acpiphp: Slot [8] registered [ 0.303129] acpiphp: Slot [9] registered [ 0.305108] acpiphp: Slot [10] registered [ 0.307118] acpiphp: Slot [11] registered [ 0.309149] acpiphp: Slot [12] registered [ 0.311205] acpiphp: Slot [13] registered [ 0.312113] acpiphp: Slot [14] registered [ 0.314096] acpiphp: Slot [15] registered [ 0.316103] acpiphp: Slot [16] registered [ 0.318096] acpiphp: Slot [17] registered [ 0.320138] acpiphp: Slot [18] registered [ 0.322102] acpiphp: Slot [19] registered [ 0.323136] acpiphp: Slot [20] registered [ 0.324080] acpiphp: Slot [21] registered [ 0.327162] acpiphp: Slot [22] registered [ 0.328087] acpiphp: Slot [23] registered [ 0.329151] acpiphp: Slot [24] registered [ 0.331074] acpiphp: Slot [25] registered [ 0.332073] acpiphp: Slot [26] registered [ 0.333082] acpiphp: Slot [27] registered [ 0.334102] acpiphp: Slot [28] registered [ 0.336079] acpiphp: Slot [29] registered [ 0.337085] acpiphp: Slot [30] registered [ 0.338077] acpiphp: Slot [31] registered [ 0.339053] PCI host bridge to bus 0000:00 [ 0.340016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356019] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.359490] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366586] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.370408] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378665] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383038] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387699] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.402018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413308] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457783] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.490013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.501019] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.504461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.508421] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.511382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.514217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.515166] iommu: Default domain type: Passthrough [ 0.517290] SCSI subsystem initialized [ 0.519111] ACPI: bus type USB registered [ 0.520218] usbcore: registered new interface driver usbfs [ 0.523080] usbcore: registered new interface driver hub [ 0.525083] usbcore: registered new device driver usb [ 0.527190] pps_core: LinuxPPS API ver. 1 registered [ 0.530011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.534049] PTP clock support registered [ 0.535441] EDAC MC: Ver: 3.0.0 [ 0.536627] PCI: Using ACPI for IRQ routing [ 0.537614] NetLabel: Initializing [ 0.539014] NetLabel: domain hash size = 128 [ 0.541012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543165] NetLabel: unlabeled traffic allowed by default [ 0.546196] vgaarb: loaded [ 0.547360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.550019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.557000] clocksource: Switched to clocksource kvm-clock [ 0.667731] VFS: Disk quotas dquot_6.6.0 [ 0.669546] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.671763] *** VALIDATE ramfs *** [ 0.673243] *** VALIDATE hugetlbfs *** [ 0.674938] pnp: PnP ACPI init [ 0.677372] pnp: PnP ACPI: found 6 devices [ 0.692493] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.695898] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698401] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.699860] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.703199] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705896] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708569] NET: Registered protocol family 2 [ 0.711481] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.717049] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.721671] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726780] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730589] TCP: Hash tables configured (established 65536 bind 65536) [ 0.733785] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.737140] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.740163] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.743415] NET: Registered protocol family 1 [ 0.746810] RPC: Registered named UNIX socket transport module. [ 0.748983] RPC: Registered udp transport module. [ 0.750778] RPC: Registered tcp transport module. [ 0.752413] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.754860] NET: Registered protocol family 44 [ 0.756284] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.757849] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.759546] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.761091] PCI: CLS 0 bytes, default 64 [ 0.762299] Unpacking initramfs... [ 2.188329] debug: unmapping init [mem 0xffff9f883cc64000-0xffff9f883ffcffff] [ 2.192452] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.194580] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 2.197358] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.704631] Initialise system trusted keyrings [ 2.706602] Key type blacklist registered [ 2.708761] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.718250] zbud: loaded [ 2.721511] *** VALIDATE nfs *** [ 2.722740] *** VALIDATE nfs4 *** [ 2.724694] pstore: using deflate compression [ 2.728366] Platform Keyring initialized [ 2.839505] NET: Registered protocol family 38 [ 2.842400] Key type asymmetric registered [ 2.845538] Asymmetric key parser 'x509' registered [ 2.848623] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.853087] io scheduler mq-deadline registered [ 2.854781] io scheduler kyber registered [ 2.856683] io scheduler bfq registered [ 2.860367] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.870799] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.874907] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.878653] ACPI: Power Button [PWRF] [ 2.884536] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.891609] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.902713] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.929684] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.959171] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.966521] Non-volatile memory driver v1.3 [ 2.968381] Linux agpgart interface v0.103 [ 2.997599] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.000758] vda: detected capacity change from 0 to 74829824 [ 3.016389] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.019811] vdb: detected capacity change from 0 to 1073741824 [ 3.030496] libphy: Fixed MDIO Bus: probed [ 3.035321] usbcore: registered new interface driver usbserial_generic [ 3.037909] usbserial: USB Serial support registered for generic [ 3.040622] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.045396] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.047419] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.049846] mousedev: PS/2 mouse device common for all mice [ 3.053882] rtc_cmos 00:05: RTC can wake from S4 [ 3.058222] rtc_cmos 00:05: registered as rtc0 [ 3.058244] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.059479] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.066620] intel_pstate: CPU model not supported [ 3.071065] hid: raw HID events driver (C) Jiri Kosina [ 3.074478] usbcore: registered new interface driver usbhid [ 3.076776] usbhid: USB HID core driver [ 3.077573] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.081949] drop_monitor: Initializing network drop monitor service [ 3.089930] Initializing XFRM netlink socket [ 3.092381] NET: Registered protocol family 10 [ 3.095343] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.096363] Segment Routing with IPv6 [ 3.100510] NET: Registered protocol family 17 [ 3.102693] mpls_gso: MPLS GSO support [ 3.106775] RAS: Correctable Errors collector initialized. [ 3.108717] AVX version of gcm_enc/dec engaged. [ 3.110312] AES CTR mode by8 optimization enabled [ 3.183682] sched_clock: Marking stable (3183663169, 0)->(4143048400, -959385231) [ 3.189065] registered taskstats version 1 [ 3.191210] Loading compiled-in X.509 certificates [ 3.193578] zswap: loaded using pool lzo/zbud [ 3.223927] Key type big_key registered [ 3.236490] Key type encrypted registered [ 3.237838] ima: No TPM chip found, activating TPM-bypass! [ 3.239777] ima: Allocated hash algorithm: sha1 [ 3.241351] ima: No architecture policies found [ 3.242887] evm: Initialising EVM extended attributes: [ 3.244444] evm: security.selinux [ 3.245479] evm: security.ima [ 3.246414] evm: security.capability [ 3.247521] evm: HMAC attrs: 0x1 [ 3.249575] rtc_cmos 00:05: setting system clock to 2026-08-26 01:01:10 UTC (1787706070) [ 3.255633] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.258606] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.267200] Write protecting the kernel read-only data: 28672k [ 3.270987] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.274307] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.307875] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.317954] systemd[1]: Detected virtualization kvm. [ 3.319857] systemd[1]: Detected architecture x86-64. [ 3.321747] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.347652] systemd[1]: No hostname configured. [ 3.349389] systemd[1]: Set hostname to . [ 3.352159] random: systemd: uninitialized urandom read (16 bytes read) [ 3.355192] systemd[1]: Initializing machine ID from random generator. [ 3.409679] random: ln: uninitialized urandom read (6 bytes read) [ 3.499280] random: systemd: uninitialized urandom read (16 bytes read) [ 3.503476] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.514529] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.523881] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.114123] device-mapper: uevent: version 1.0.3 [ 4.116184] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.854888] random: fast init done [ 4.867950] virtio_net virtio0 ens2: renamed from eth0 [ 4.948934] scsi host0: ata_piix [ 4.993868] scsi host1: ata_piix [ 4.995666] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.998295] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.754872] random: crng init done [ 9.756337] random: 7 urandom warning(s) missed due to ratelimiting [ 9.794945] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.576032] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.018631] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.355907] SELinux: Disabled at runtime. [ 12.420758] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.430569] systemd[1]: Detected virtualization kvm. [ 12.432099] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.058344] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.062760] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.075557] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.082954] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.086208] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.096366] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.107191] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watc[ 13.217428] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS h. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.906884] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.368161] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.384288] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.545969] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.576921] EDAC sbridge: Ver: 1.1.2 [ 16.259712] Key type dns_resolver registered [ 16.587281] NFS: Registering the id_resolver key type [ 16.589590] Key type id_resolver registered [ 16.591318] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg458-client login: [ 72.917268] libcfs: loading out-of-tree module taints kernel. [ 73.022796] Key type ._llcrypt registered [ 73.029230] Key type .llcrypt registered [ 73.584820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 73.596755] alg: No test for adler32 (adler32-zlib) [ 75.282381] Lustre: Lustre: Build Version: 2.17.57_80_gcb5c033 [ 76.413279] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 78.239228] Key type lgssc registered [ 80.425403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 211.312103] hrtimer: interrupt took 3394401 ns [ 218.028633] Lustre: Mounted lustre-client - version 2.17.57_80_gcb5c033 [ 222.766541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.151477] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing check_logdir /tmp/testlogs/ [ 239.766074] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing yml_node [ 243.680620] Lustre: lustre-OST0000-osc-ffff9f88853bd000: disconnect after 23s idle [ 243.978542] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 246.626861] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 249.407831] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 251.192924] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 25 21:05:16 EDT 2026 [ 267.840479] Lustre: Unmounted lustre-client [ 304.974752] Key type lgssc unregistered [ 305.280170] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.292265] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.320507] LNet: Removed LNI 192.168.204.58@tcp [ 306.223279] Key type .llcrypt unregistered [ 306.226928] Key type ._llcrypt unregistered [ 334.831426] Key type ._llcrypt registered [ 334.833860] Key type .llcrypt registered [ 335.046850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.064025] alg: No test for adler32 (adler32-zlib) [ 336.008524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 336.086977] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 336.111615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 340.538895] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 340.687546] LNet: 9303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.695673] LNetError: 9303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.712562] LNet: Removed LNI 192.168.204.58@tcp [ 341.438635] Key type .llcrypt unregistered [ 341.442242] Key type ._llcrypt unregistered [ 347.911857] Key type ._llcrypt registered [ 347.914792] Key type .llcrypt registered [ 349.208616] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:06:54 (1787706414) [ 349.644370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.664479] alg: No test for adler32 (adler32-zlib) [ 350.609662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.746281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.798887] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.825751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.879580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 350.907502] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 358.197170] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:07:03 (1787706423) [ 358.318719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.360741] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.403361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.532685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 367.037856] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:07:12 (1787706432) [ 367.189119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.249223] LNet: 10887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.337305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.450337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 374.676486] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:07:20 (1787706440) [ 374.973062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.034938] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.081966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.188245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 382.312576] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:07:28 (1787706448) [ 382.379522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.408662] LNet: 12056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.433207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.528647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 382.794747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 389.274103] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:07:35 (1787706455) [ 389.858596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.936933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.100839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.655080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 391.023333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 399.203448] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:07:45 (1787706465) [ 399.462608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.481614] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.486517] LNet: 13250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 399.506620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.568785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 406.733048] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:07:52 (1787706472) [ 407.087926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.152314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.247522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 407.290684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.906491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 407.965160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.642712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 408.703921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 409.104544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 409.132886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 409.504200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 409.544683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 410.036647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 410.072323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.380040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 410.416992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.058104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 411.087960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 411.701953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 411.744103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 412.275468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 412.305681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.674622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 412.724493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 413.323855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 413.358492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.788894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 413.848332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 421.269560] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:08:07 (1787706487) [ 421.849496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.898536] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.909162] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 421.932783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.068749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 422.103552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 429.198359] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:08:15 (1787706495) [ 429.686601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.736406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.804173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.855634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.435240] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:08:23 (1787706503) [ 437.629687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.746911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.813812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 437.851299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.358353] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:08:30 (1787706510) [ 444.544776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.573513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.651520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.680239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 451.257219] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:08:37 (1787706517) [ 451.437527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.505227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.601758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 451.659950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 458.024286] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:08:43 (1787706523) [ 458.120250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.139102] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.143617] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 458.166779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.255659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 458.289142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 464.326022] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:08:50 (1787706530) [ 464.532796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.581446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.675389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.108194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.492529] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:08:58 (1787706538) [ 473.089478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.147976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.439559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.719803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.255704] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:09:07 (1787706547) [ 482.604357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.660381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.798108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.041442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.302453] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:09:16 (1787706556) [ 491.459964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.513581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.127960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.485974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.075478] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:09:25 (1787706565) [ 500.367718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.457334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.575765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.642628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 508.616774] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:09:34 (1787706574) [ 508.779756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.844312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.973821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.015772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.405222] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:09:42 (1787706582) [ 516.761805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.822982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.908946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 524.079138] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:09:50 (1787706590) [ 524.438991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.485290] LNet: 22303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.488268] LNet: 22303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 524.498446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.579522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 531.673598] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:09:57 (1787706597) [ 532.021371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.088440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.228879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 532.593504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 539.213186] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:10:05 (1787706605) [ 539.558384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.633423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.756375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 539.793746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 548.968141] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:10:14 (1787706614) [ 549.054877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.095853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.141904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 549.170662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 556.031786] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:10:22 (1787706622) [ 556.303254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.338602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.390866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 556.415257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 562.370099] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:10:28 (1787706628) [ 562.683994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.706939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.762949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 562.796535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 569.438805] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:10:35 (1787706635) [ 569.562368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.625680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.664924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 569.717543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 569.736848] LNetError: 25837:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 569.771036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 569.800277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 575.805226] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:10:41 (1787706641) [ 576.194651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.230942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.381316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4n560czKnj [ 576.463407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4n560czKnj [ 576.526860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4n560czKnj [ 582.158290] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:10:48 (1787706648) [ 582.345344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.397598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.465631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 582.523697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 582.597349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 588.437966] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:10:54 (1787706654) [ 588.712972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.781234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.889193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 588.922413] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 590.469759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 592.113889] Lustre: Lustre: Build Version: 2.17.57_80_gcb5c033 [ 594.247251] Key type lgssc registered [ 595.771155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 600.151173] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.58@tcp99 has no available nets [ 603.212217] Key type lgssc unregistered [ 603.573247] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.646430] LNet: Removed LNI 192.168.204.58@tcp [ 605.416129] Key type .llcrypt unregistered [ 605.419851] Key type ._llcrypt unregistered [ 612.107291] Key type ._llcrypt registered [ 612.110125] Key type .llcrypt registered [ 613.428492] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:11:19 (1787706679) [ 614.163496] Key type .llcrypt unregistered [ 614.166964] Key type ._llcrypt unregistered [ 621.418426] Key type ._llcrypt registered [ 621.421059] Key type .llcrypt registered [ 621.577923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.595183] alg: No test for adler32 (adler32-zlib) [ 622.593077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.658967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.685031] LNet: 29630:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 622.690374] LNet: 29630:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 622.694441] LNet: 29630:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 622.703258] LNet: Message forwarding will be enabled in 2s [ 625.759178] LNet: Message forwarding enabled [ 626.115673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.149365] LNet: Message forwarding will stop in 2s to 6s [ 629.858737] LNet: Message forwarding disabled [ 630.630466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.677390] LNet: 29729:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.687821] LNet: 29729:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.697612] LNet: 29729:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.710495] LNet: Message forwarding will be enabled in 2s [ 630.839271] LNet: 29742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.487754] Key type .llcrypt unregistered [ 631.495527] Key type ._llcrypt unregistered [ 643.451956] Key type ._llcrypt registered [ 643.454610] Key type .llcrypt registered [ 644.566203] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:11:50 (1787706710) [ 645.272631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.291644] alg: No test for adler32 (adler32-zlib) [ 646.227420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.299936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 652.149743] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:11:58 (1787706718) [ 652.307540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.345680] LNet: 31150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.373471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.455894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 652.503994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 652.550643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 652.589559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 652.625576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 652.655936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 652.689866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 652.724478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 652.764979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 652.804176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 652.842444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 652.900617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 652.949741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 652.987615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 653.032928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 653.074870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 653.116797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 653.158761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 653.199979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 653.220783] LNetError: 31193:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 658.956859] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:12:04 (1787706724) [ 659.173647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.206452] LNet: 31762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.268522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.429798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.614596] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:12:12 (1787706732) [ 668.932411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg458-server [ 671.104788] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:12:16 (1787706736) [ 671.184756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.212104] LNet: 32689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.245144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.355698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.389828] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 672.417257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[59-63]@tcp [ 672.475654] LNet: 32830:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 672.541378] LNetError: 32698:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 672.561103] LNetError: 32698:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.59@tcp has gone from up to down [ 675.554897] LNetError: 32706:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.61@tcp has gone from up to down [ 675.567696] LNetError: 32706:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 679.929082] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:12:25 (1787706745) [ 681.755505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg458-server [ 683.803284] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:12:29 (1787706749) [ 683.981910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.028366] LNet: 33766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.038859] LNetError: 33766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.065752] LNet: Removed LNI 192.168.204.58@tcp [ 684.098890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.187608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.221527] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 685.035362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[59-64/2]@tcp [ 685.058214] LNet: 33884:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 685.064365] LNet: 33884:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 685.090460] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.59@tcp is being used as a gateway but routing feature is not turned on [ 685.092407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[59-64/2]@tcp [ 685.102855] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 685.118358] LNetError: 33770:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.59@tcp has gone from up to down [ 685.128676] LNetError: 33770:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 692.120917] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:12:37 (1787706757) [ 692.382716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.403346] LNet: 34476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.412104] LNetError: 34476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.438513] LNet: Removed LNI 192.168.204.58@tcp [ 692.464238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.560756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 692.625710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.661295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.786323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.860622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.909835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.951245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 693.083359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.159876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.260899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.357941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.412406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.487706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 693.616466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.675725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.761299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.866700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.934166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.014494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 694.141677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.212412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.307364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.419970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.457331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.502653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 694.583392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.615653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.684314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.776073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.808867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.853282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 694.917619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.959369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.062450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.274667] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:12:47 (1787706767) [ 703.194835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg458-server [ 705.132864] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:12:50 (1787706770) [ 707.087102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg458-server [ 708.775141] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:12:54 (1787706774) [ 709.086814] LNet: 35878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.093440] LNet: 35878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 709.848208] Key type .llcrypt unregistered [ 709.856642] Key type ._llcrypt unregistered [ 717.102541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 717.133688] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.136203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 717.138906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 719.808964] Key type ._llcrypt registered [ 719.811411] Key type .llcrypt registered [ 720.045348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 720.062233] alg: No test for adler32 (adler32-zlib) [ 721.055686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.173082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.260457] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 722.362407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.395436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.067197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.092633] LNet: Removed LNI 10.1.2.3@tcp [ 725.286908] LNet: 36602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.309626] LNetError: 36602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.341759] LNet: Removed LNI 192.168.204.58@tcp [ 725.891604] Key type .llcrypt unregistered [ 725.894253] Key type ._llcrypt unregistered [ 740.658541] Key type ._llcrypt registered [ 740.661191] Key type .llcrypt registered [ 742.524696] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:13:27 (1787706807) [ 743.259677] Key type .llcrypt unregistered [ 743.265414] Key type ._llcrypt unregistered [ 750.586705] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.798881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.801220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.814224] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.436120] Key type ._llcrypt registered [ 753.439853] Key type .llcrypt registered [ 753.634621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.647533] alg: No test for adler32 (adler32-zlib) [ 754.518540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.582455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.600352] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 755.184761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 755.202101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.698836] LNet: 38107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.704599] LNetError: 38107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 757.713904] LNet: Removed LNI 10.1.2.3@tcp [ 757.722152] LNet: Removed LNI 192.168.204.58@tcp [ 758.140431] Key type .llcrypt unregistered [ 758.144888] Key type ._llcrypt unregistered [ 770.488491] Key type ._llcrypt registered [ 770.494230] Key type .llcrypt registered [ 772.158821] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:13:57 (1787706837) [ 772.945298] Key type .llcrypt unregistered [ 772.949620] Key type ._llcrypt unregistered [ 781.048838] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.323419] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.346574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.364531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.055563] Key type ._llcrypt registered [ 784.057257] Key type .llcrypt registered [ 784.220199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.239287] alg: No test for adler32 (adler32-zlib) [ 785.176753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.228278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 785.246270] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 787.936623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.954326] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.435439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 790.473133] LNet: Removed LNI 10.1.2.3@tcp [ 790.485201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 790.532872] LNet: Removed LNI 10.1.2.31@tcp [ 790.551413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 790.583647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.686733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 791.715063] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 792.787806] LNet: 40031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.800866] LNetError: 40031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 792.821061] LNet: Removed LNI 10.1.2.31@tcp [ 792.844668] LNet: Removed LNI 10.1.2.3@tcp [ 793.672692] Key type .llcrypt unregistered [ 793.677429] Key type ._llcrypt unregistered [ 810.262695] Key type ._llcrypt registered [ 810.265656] Key type .llcrypt registered [ 811.799579] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:14:37 (1787706877) [ 813.210572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.262341] alg: No test for adler32 (adler32-zlib) [ 814.233869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.324650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.342507] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 815.482878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.562827] LNet: 40990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.586714] LNetError: 40990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.600662] LNet: Removed LNI 192.168.204.58@tcp [ 815.624356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.686759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.58@tcp [ 815.711732] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 816.546957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.58@tcp [ 816.627102] LNet: Removed LNI 192.168.204.58@tcp [ 816.770839] LNet: 41125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.973901] Key type .llcrypt unregistered [ 817.976432] Key type ._llcrypt unregistered [ 830.909803] Key type ._llcrypt registered [ 830.912523] Key type .llcrypt registered [ 832.234754] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:14:57 (1787706897) [ 834.500369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg458-server [ 836.237653] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:15:02 (1787706902) [ 836.737884] Key type .llcrypt unregistered [ 836.741076] Key type ._llcrypt unregistered [ 843.004352] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.222858] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.229300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.233197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.927573] Key type ._llcrypt registered [ 845.932471] Key type .llcrypt registered [ 846.152122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.166650] alg: No test for adler32 (adler32-zlib) [ 847.107805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.153337] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 847.725167] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 849.848276] LNet: 42969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.867204] LNetError: 42969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.877045] LNet: Removed LNI 192.168.204.58@tcp1 [ 849.885331] LNet: Removed LNI 10.1.2.3@tcp2 [ 850.626051] Key type .llcrypt unregistered [ 850.629525] Key type ._llcrypt unregistered [ 862.392515] Key type ._llcrypt registered [ 862.396846] Key type .llcrypt registered [ 864.249491] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:15:29 (1787706929) [ 864.916741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.942283] alg: No test for adler32 (adler32-zlib) [ 865.869609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.898453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.911986] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 866.773187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.794821] LNet: 43941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.801686] LNetError: 43941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.817615] LNet: Removed LNI 192.168.204.58@tcp [ 866.834132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.903262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 866.935523] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 873.766942] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:15:39 (1787706939) [ 873.917631] LNet: 44524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.934579] LNetError: 44524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.962144] LNet: Removed LNI 192.168.204.58@tcp [ 874.924575] Key type .llcrypt unregistered [ 874.933167] Key type ._llcrypt unregistered [ 882.510501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 882.684547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 882.690598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 882.696480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.298511] Key type ._llcrypt registered [ 885.300618] Key type .llcrypt registered [ 885.529751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.565459] alg: No test for adler32 (adler32-zlib) [ 886.491320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.564629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.592766] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 887.319668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 887.344703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.689678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.716152] LNet: 45279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.742545] LNetError: 45279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 889.756951] LNet: Removed LNI 10.1.2.3@tcp [ 889.775246] LNet: Removed LNI 192.168.204.58@tcp [ 889.805980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.928134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 889.950493] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 889.956121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 890.494685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.526641] LNet: 45307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.535706] LNetError: 45307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 890.546300] LNetError: 45307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 890.553065] LNet: Removed LNI 10.1.2.3@tcp [ 890.568192] LNet: Removed LNI 192.168.204.58@tcp [ 890.601294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.687502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 890.718310] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 890.729617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 891.146227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 891.178683] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 892.232173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.267855] LNet: 45469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.283599] LNetError: 45469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.288775] LNetError: 45469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 892.294972] LNet: Removed LNI 10.1.2.3@tcp [ 892.300335] LNet: Removed LNI 192.168.204.58@tcp [ 892.314719] LNet: Removed LNI 192.168.204.58@tcp2 [ 892.343551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.428259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.469272] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 892.484746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.494285] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 901.405670] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:16:07 (1787706967) [ 901.724906] LNet: 46068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.740349] LNetError: 46068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.752896] LNetError: 46068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 901.773081] LNet: Removed LNI 10.1.2.3@tcp [ 901.775907] LNet: Removed LNI 192.168.204.58@tcp [ 901.784366] LNet: Removed LNI 192.168.204.58@tcp2 [ 902.535980] Key type .llcrypt unregistered [ 902.538328] Key type ._llcrypt unregistered [ 909.478408] Key type ._llcrypt registered [ 909.481705] Key type .llcrypt registered [ 909.816880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 909.834656] alg: No test for adler32 (adler32-zlib) [ 910.906711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.998382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.061342] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 911.903602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.933710] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.941126] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.962856] LNet: Removed LNI 192.168.204.58@tcp [ 911.978158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.050026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.086199] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 912.786154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.822576] LNet: 46498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.836124] LNetError: 46498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.865863] LNet: Removed LNI 192.168.204.58@tcp [ 912.892224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.997459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.027621] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 913.777251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.807636] LNet: Removed LNI 192.168.204.58@tcp [ 913.854647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.904033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.937428] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 914.695437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.869469] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.876131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.884225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.476235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.524682] LNet: 46607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.535560] LNet: 46607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 917.545198] LNetError: 46607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.555235] LNetError: 46607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 917.576373] LNet: Removed LNI 192.168.204.58@tcp [ 917.621265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.737018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.803466] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 924.675662] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:16:30 (1787706990) [ 924.990392] LNet: 47205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.998430] LNetError: 47205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.015697] LNet: Removed LNI 192.168.204.58@tcp [ 925.954883] Key type .llcrypt unregistered [ 925.960929] Key type ._llcrypt unregistered [ 932.559387] Key type ._llcrypt registered [ 932.565418] Key type .llcrypt registered [ 932.839190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.868992] alg: No test for adler32 (adler32-zlib) [ 933.791975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.855232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.919960] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 934.908338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.947284] LNet: 47607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.959110] LNetError: 47607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.981915] LNet: Removed LNI 192.168.204.58@tcp [ 935.004258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.116940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 935.165046] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 942.742290] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:16:48 (1787707008) [ 942.895821] LNet: 48191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.908799] LNetError: 48191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.927307] LNet: Removed LNI 192.168.204.58@tcp [ 943.675820] Key type .llcrypt unregistered [ 943.680946] Key type ._llcrypt unregistered [ 949.647082] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.734582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.743312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.749765] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 952.365420] Key type ._llcrypt registered [ 952.368281] Key type .llcrypt registered [ 952.642451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.668295] alg: No test for adler32 (adler32-zlib) [ 953.630786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.707694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.735203] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 954.667425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.710476] LNet: 48653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.719084] LNetError: 48653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.746812] LNet: Removed LNI 192.168.204.58@tcp [ 954.790027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.875396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 954.914844] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 963.226619] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:17:08 (1787707028) [ 963.352859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.382400] LNet: 49244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.401943] LNetError: 49244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.424108] LNet: Removed LNI 192.168.204.58@tcp [ 963.475571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.654495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 971.606829] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:17:17 (1787707037) [ 972.053277] LNet: 49836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.966832] Key type .llcrypt unregistered [ 972.970322] Key type ._llcrypt unregistered [ 978.656365] Key type ._llcrypt registered [ 978.669893] Key type .llcrypt registered [ 978.902037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.940924] alg: No test for adler32 (adler32-zlib) [ 979.898874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.973743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.994112] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 980.840598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.856534] LNet: 50237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.862668] LNetError: 50237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.876878] LNet: Removed LNI 192.168.204.58@tcp [ 980.900793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.974536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 980.998700] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 987.203972] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:17:33 (1787707053) [ 987.362609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.412644] LNet: 50829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.421581] LNetError: 50829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.451729] LNet: Removed LNI 192.168.204.58@tcp [ 987.487613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.547511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 994.956536] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:17:40 (1787707060) [ 996.713519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 998.368739] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:17:44 (1787707064) [ 1000.025132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1001.934796] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:17:47 (1787707067) [ 1003.704535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1005.482478] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:17:51 (1787707071) [ 1007.057595] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1008.716916] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:17:54 (1787707074) [ 1010.020237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1011.818063] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:17:57 (1787707077) [ 1012.028015] LNet: 52884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.753741] Key type .llcrypt unregistered [ 1012.756533] Key type ._llcrypt unregistered [ 1019.130749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.263242] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.273663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.280688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.857214] Key type ._llcrypt registered [ 1021.860039] Key type .llcrypt registered [ 1022.073050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.103636] alg: No test for adler32 (adler32-zlib) [ 1023.082707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.152673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.167963] LNet: Added LNI 192.168.204.58@tcp [8/512/0/180] [ 1023.172307] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1023.175352] LNet: 53211:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.252333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.279852] LNet: 53230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.289201] LNetError: 53230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.301100] LNet: Removed LNI 10.1.2.3@tcp [ 1023.307376] LNetError: 53230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.321785] LNet: Removed LNI 192.168.204.58@tcp [ 1023.361357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.456125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.521030] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1023.614609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.635368] LNet: 53254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.641904] LNetError: 53254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.653863] LNet: Removed LNI 192.168.204.58@tcp [ 1023.673030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.762910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.790794] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1023.798985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.801692] LNet: 53262:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.899732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.918656] LNet: 53280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.923627] LNetError: 53280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.927560] LNet: Removed LNI 10.1.2.3@tcp [ 1023.931780] LNetError: 53280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.952186] LNet: Removed LNI 192.168.204.58@tcp [ 1023.970242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.037763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.057926] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1024.062750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.064403] LNet: 53288:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.321755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.356219] LNet: 53314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.361965] LNetError: 53314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.367721] LNet: Removed LNI 10.1.2.3@tcp [ 1024.370139] LNetError: 53314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.387535] LNet: Removed LNI 192.168.204.58@tcp [ 1024.408802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.509980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.540522] LNet: Added LNI 192.168.204.58@tcp [8/256/0/182] [ 1024.554479] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1024.708784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.742666] LNet: 53343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.750648] LNetError: 53343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.756940] LNet: Removed LNI 10.1.2.3@tcp [ 1024.760923] LNetError: 53343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.774732] LNet: Removed LNI 192.168.204.58@tcp [ 1024.804678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.883786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.906937] LNet: Added LNI 192.168.204.58@tcp [8/256/0/182] [ 1024.910834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1024.914771] LNet: 53351:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.986442] LNet: Removed LNI 10.1.2.3@tcp [ 1025.021235] LNet: Removed LNI 192.168.204.58@tcp [ 1025.747094] Key type .llcrypt unregistered [ 1025.748686] Key type ._llcrypt unregistered [ 1038.180713] Key type ._llcrypt registered [ 1038.182827] Key type .llcrypt registered [ 1039.769874] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:18:25 (1787707105) [ 1041.483289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1043.216409] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:18:29 (1787707109) [ 1043.738953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1043.754810] alg: No test for adler32 (adler32-zlib) [ 1044.676692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.759526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1044.788238] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1045.587507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.58 [ 1045.611702] LNet: 54619:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1045.620095] LNet: Removed LNI 192.168.204.58@tcp [ 1051.910411] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:18:37 (1787707117) [ 1052.334576] LNet: 55189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.356029] LNetError: 55189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.384606] LNet: Removed LNI 192.168.204.58@tcp [ 1053.019461] Key type .llcrypt unregistered [ 1053.022789] Key type ._llcrypt unregistered [ 1060.363197] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1060.597768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1063.446524] Key type ._llcrypt registered [ 1063.448447] Key type .llcrypt registered [ 1063.603113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.619809] alg: No test for adler32 (adler32-zlib) [ 1065.009844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.792397] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:18:57 (1787707137) [ 1071.929407] LNet: 56100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.940816] LNetError: 56100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1071.959990] LNet: Removed LNI 10.1.2.3@tcp [ 1072.796694] Key type .llcrypt unregistered [ 1072.802155] Key type ._llcrypt unregistered [ 1080.451667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1080.671522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1080.674269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1080.676881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1083.429383] Key type ._llcrypt registered [ 1083.433130] Key type .llcrypt registered [ 1083.656250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.683315] alg: No test for adler32 (adler32-zlib) [ 1084.710361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1091.632944] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:19:17 (1787707157) [ 1093.277829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1095.170705] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:19:21 (1787707161) [ 1095.635829] LNet: 57294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.646483] LNetError: 57294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.661648] LNet: Removed LNI 10.1.2.3@tcp [ 1096.572626] Key type .llcrypt unregistered [ 1096.580875] Key type ._llcrypt unregistered [ 1103.787237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1104.058868] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1104.062415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1104.068529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1106.763482] Key type ._llcrypt registered [ 1106.766745] Key type .llcrypt registered [ 1106.882348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.892880] alg: No test for adler32 (adler32-zlib) [ 1107.780198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.845392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1117.034642] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:19:42 (1787707182) [ 1117.472141] LNet: 58528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.486660] LNetError: 58528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.511108] LNet: Removed LNI 10.1.2.3@tcp [ 1118.307309] Key type .llcrypt unregistered [ 1118.308807] Key type ._llcrypt unregistered [ 1126.178690] Key type ._llcrypt registered [ 1126.181707] Key type .llcrypt registered [ 1126.411304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.429108] alg: No test for adler32 (adler32-zlib) [ 1127.313755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1127.338074] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1135.864843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1136.751504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_interrupt -r 1 [ 1136.825809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_interrupt -r 1 [ 1136.928850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1136.964964] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1137.106318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.128385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.149671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.173589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.193839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.235447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.275133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.311380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.330264] LNet: 58970:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.355817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.392827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.432306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.483227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.524615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.567492] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1137.575245] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1137.599897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.652603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.330773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_dropped -r 1 [ 1138.370754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_dropped -r 1 [ 1138.429746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1138.728242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.766947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.830178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.880627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.955973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.020343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.069440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.099536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.133505] LNet: 59080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.166966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.252702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.319588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.372834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.411222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.452719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.481462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.123333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_aborted -r 1 [ 1140.194061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_aborted -r 1 [ 1140.226139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1140.240930] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1140.247479] LNet: Skipped 1 previous similar message [ 1140.543685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.606202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.646820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.668814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.689917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.711611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.749240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.776672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.792977] LNet: 59189:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.806748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.847499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.886089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.924263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.977391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.011562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.051233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.702640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_no_route -r 1 [ 1141.754559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_no_route -r 1 [ 1141.807742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1142.163149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.206300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.264603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.325453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.365988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.443924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.538958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.595501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.721637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.831263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.898369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.986810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.046297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.114078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.141468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.702507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_timeout -r 1 [ 1143.733812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_timeout -r 1 [ 1143.773405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1143.786775] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1143.790596] LNet: Skipped 1 previous similar message [ 1143.999188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.031057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.060432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.079443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.100219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.137628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.160579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.189573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.205970] LNet: 59408:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.214381] LNet: 59408:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1144.227443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.257522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.315343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.388384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.423826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.455760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.500018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.947477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_error -r 1 [ 1144.985655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_error -r 1 [ 1145.024407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1145.293967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.347870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.402534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.452438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.473537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.491385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.517875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.554088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.592990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.644210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.694690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.729123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.786728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.853747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.889629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.337043] LNet: 59562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.353401] LNetError: 59562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.396250] LNet: Removed LNI 192.168.204.58@tcp [ 1146.651731] Key type .llcrypt unregistered [ 1146.653313] Key type ._llcrypt unregistered [ 1169.654527] Key type ._llcrypt registered [ 1169.659414] Key type .llcrypt registered [ 1171.448207] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:20:36 (1787707236) [ 1172.577584] Key type .llcrypt unregistered [ 1172.580677] Key type ._llcrypt unregistered [ 1181.296241] Key type ._llcrypt registered [ 1181.303977] Key type .llcrypt registered [ 1181.710723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1181.752813] alg: No test for adler32 (adler32-zlib) [ 1182.776152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1182.830272] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1191.070692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1203.974498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.059913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1204.088381] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1205.630869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_interrupt -r 1 [ 1205.709547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_interrupt -r 1 [ 1205.769588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_interrupt -r 1 [ 1205.813620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_interrupt -r 1 [ 1205.872871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_interrupt -r 1 [ 1205.912946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_interrupt -r 1 [ 1205.941683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_interrupt -r 1 [ 1205.967563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_interrupt -r 1 [ 1206.003281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1206.036900] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1206.047020] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1206.341511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.396526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.432369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.480155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.556584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.620027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.686783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.785927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.824714] LNet: 61035:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.887772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.975049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.039844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.072374] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1207.079068] LNet: Skipped 3 previous similar messages [ 1207.115354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.175595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.252336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.302851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.079575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_dropped -r 1 [ 1208.118709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_dropped -r 1 [ 1208.155902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_dropped -r 1 [ 1208.198210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_dropped -r 1 [ 1208.233129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_dropped -r 1 [ 1208.267734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_dropped -r 1 [ 1208.309169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_dropped -r 1 [ 1208.344660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_dropped -r 1 [ 1208.392957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1208.427020] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1208.779226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.836822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.881277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.927842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.989271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.067245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.164490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.201365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.220283] LNet: 61157:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.245899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.287942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.331969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.413299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.479876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.535877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.621621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.664475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_aborted -r 1 [ 1210.729030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_aborted -r 1 [ 1210.812176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_aborted -r 1 [ 1210.923611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_aborted -r 1 [ 1211.050639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_aborted -r 1 [ 1211.176155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_aborted -r 1 [ 1211.225812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_aborted -r 1 [ 1211.281584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_aborted -r 1 [ 1211.360445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1211.396392] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1211.408593] LNet: Skipped 2 previous similar messages [ 1211.870528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.918413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.972988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.025156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.062143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.125303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.213965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.285669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.321661] LNet: 61278:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.391650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.465728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.564243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.648718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.752751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.871656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.932698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.840282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_no_route -r 1 [ 1213.885897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_no_route -r 1 [ 1213.918186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_no_route -r 1 [ 1213.946588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_no_route -r 1 [ 1214.010429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_no_route -r 1 [ 1214.086157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_no_route -r 1 [ 1214.139840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_no_route -r 1 [ 1214.191120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_no_route -r 1 [ 1214.287977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1214.581602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.630423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.655354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.695666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.745477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.790724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.817389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.860631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.889521] LNet: 61400:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.921581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.965370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.011575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.085077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.156833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.233352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.284839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.293907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_timeout -r 1 [ 1216.365733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_timeout -r 1 [ 1216.433137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_timeout -r 1 [ 1216.504830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_timeout -r 1 [ 1216.598260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_timeout -r 1 [ 1216.671444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_timeout -r 1 [ 1216.717177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_timeout -r 1 [ 1216.773655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_timeout -r 1 [ 1216.886954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1216.939603] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1216.948738] LNet: Skipped 5 previous similar messages [ 1217.363922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.412428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.472084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.560702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.637536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.714596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.782696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1217.838325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1217.927372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.033705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.099380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.183922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.245115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.316025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.384523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.316041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_error -r 1 [ 1219.447251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_error -r 1 [ 1219.541772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_error -r 1 [ 1219.625975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_error -r 1 [ 1219.718214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_error -r 1 [ 1219.803942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_error -r 1 [ 1219.876414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_error -r 1 [ 1219.946155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_error -r 1 [ 1220.013386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1220.462922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.524311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.577783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.629665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.674839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.738121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.844098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.931248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.972935] LNet: 61643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.982675] LNet: 61643:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1221.031558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.111654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.190731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.331663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.388313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.457339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.549076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.840360] LNet: 61723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.848502] LNetError: 61723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.857512] LNet: Removed LNI 192.168.204.58@tcp [ 1228.907662] LNet: Removed LNI 192.168.204.58@tcp1 [ 1229.240677] Key type .llcrypt unregistered [ 1229.246386] Key type ._llcrypt unregistered [ 1247.584457] Key type ._llcrypt registered [ 1247.586492] Key type .llcrypt registered [ 1248.876723] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:21:54 (1787707314) [ 1249.709280] Key type .llcrypt unregistered [ 1249.711817] Key type ._llcrypt unregistered [ 1256.465432] Key type ._llcrypt registered [ 1256.475513] Key type .llcrypt registered [ 1256.880421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1256.893676] alg: No test for adler32 (adler32-zlib) [ 1257.983286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1258.058409] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1267.411429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1268.631569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_dropped -r 1 [ 1268.679978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_dropped -r 1 [ 1268.720529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1268.752539] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1269.033526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.078561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.130758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.182422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.229747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.273251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.326525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.367614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.387029] LNet: 62976:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.411137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.461392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.500509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.542477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.583290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.619535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.651086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.568909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_error -r 1 [ 1270.650258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_error -r 1 [ 1270.741208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1270.782742] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1271.153852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.198935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.262212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.309296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.402362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.454854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.513367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.573354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.602348] LNet: 63085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.636871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.707413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.763595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.809184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.846850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.893033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1271.948635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1272.497615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_timeout -r 1 [ 1272.541017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_timeout -r 1 [ 1272.585990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1272.612054] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1272.769442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.794856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.820524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.842773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.875664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.930621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.988317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.041074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.081415] LNet: 63194:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1273.108172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.157242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.207694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.252671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.299816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.358828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.419084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1273.909849] LNet: 63240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.930244] LNetError: 63240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.967822] LNet: Removed LNI 192.168.204.58@tcp [ 1274.529957] Key type .llcrypt unregistered [ 1274.533290] Key type ._llcrypt unregistered [ 1294.701555] Key type ._llcrypt registered [ 1294.703144] Key type .llcrypt registered [ 1295.730594] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:22:41 (1787707361) [ 1296.926324] Key type .llcrypt unregistered [ 1296.928224] Key type ._llcrypt unregistered [ 1304.615679] Key type ._llcrypt registered [ 1304.618354] Key type .llcrypt registered [ 1304.801797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1304.815718] alg: No test for adler32 (adler32-zlib) [ 1305.803761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1305.833524] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1313.852892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1323.854461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.939426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.973190] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1325.334210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_dropped -r 1 [ 1325.371985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e remote_dropped -r 1 [ 1325.411702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_dropped -r 1 [ 1325.441217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e remote_dropped -r 1 [ 1325.474378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e remote_dropped -r 1 [ 1325.553650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e remote_dropped -r 1 [ 1325.608934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e remote_dropped -r 1 [ 1325.698541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e remote_dropped -r 1 [ 1325.758647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1325.793769] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1325.808560] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.158@tcp [ 1326.038871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.078644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.101588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.128917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.157108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1326.191671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1326.235745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.280656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1326.297392] LNet: 64712:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1326.312716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1326.350794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1326.382560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.418231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1326.459829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1326.525425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.561655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1326.613696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.172183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_error -r 1 [ 1327.205944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e remote_error -r 1 [ 1327.247962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_error -r 1 [ 1327.283736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e remote_error -r 1 [ 1327.323223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e remote_error -r 1 [ 1327.361822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e remote_error -r 1 [ 1327.419074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e remote_error -r 1 [ 1327.447645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e remote_error -r 1 [ 1327.474286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1327.491745] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1327.496213] LNet: Skipped 2 previous similar messages [ 1327.688567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1327.720791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1327.753565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1327.803365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1327.848161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.912258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.950214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.991549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.017462] LNet: 64835:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.034546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.084337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.138670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.188740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.271624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.333114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1328.384469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1328.421354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.117282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_timeout -r 1 [ 1329.171214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e remote_timeout -r 1 [ 1329.224428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_timeout -r 1 [ 1329.275306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e remote_timeout -r 1 [ 1329.316408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e remote_timeout -r 1 [ 1329.361472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e remote_timeout -r 1 [ 1329.402521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e remote_timeout -r 1 [ 1329.439209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e remote_timeout -r 1 [ 1329.496482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.158@tcp [ 1329.524715] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1329.840321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.889219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1329.945896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.974418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.043385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.075497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.108975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.147687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.167589] LNet: 64958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.186865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.231750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.289037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.344192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.372838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.398902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.456247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.509113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.190950] LNet: 65040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.197973] LNetError: 65040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.206577] LNet: Removed LNI 192.168.204.58@tcp [ 1336.224063] LNet: Removed LNI 192.168.204.58@tcp1 [ 1336.706680] Key type .llcrypt unregistered [ 1336.710069] Key type ._llcrypt unregistered [ 1354.057541] Key type ._llcrypt registered [ 1354.060479] Key type .llcrypt registered [ 1355.323245] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:23:41 (1787707421) [ 1356.884356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1358.523096] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:23:44 (1787707424) [ 1359.379270] Key type .llcrypt unregistered [ 1359.383373] Key type ._llcrypt unregistered [ 1365.265817] Key type ._llcrypt registered [ 1365.267387] Key type .llcrypt registered [ 1365.446653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.458506] alg: No test for adler32 (adler32-zlib) [ 1366.326332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1366.348802] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1372.960613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1373.650858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e network_timeout -r 1 [ 1373.718914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e network_timeout -r 1 [ 1373.779340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1373.808276] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1374.044697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1374.070981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1374.093453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1374.114625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1374.155794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1374.176918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1374.202402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1374.246144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1374.280953] LNet: 66609:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1374.305591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1374.368910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1374.418748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1374.472951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1374.539865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1374.570043] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1374.575256] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1374.633634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1374.992619] LNet: 66652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.997088] LNetError: 66652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.016314] LNet: Removed LNI 192.168.204.58@tcp [ 1375.503538] Key type .llcrypt unregistered [ 1375.505402] Key type ._llcrypt unregistered [ 1402.844606] Key type ._llcrypt registered [ 1402.847764] Key type .llcrypt registered [ 1403.333967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1403.354506] alg: No test for adler32 (adler32-zlib) [ 1404.420454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1404.481956] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1415.029680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1426.784830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.830533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.892394] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1428.756826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e network_timeout -r 1 [ 1428.823602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e network_timeout -r 1 [ 1428.909399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e network_timeout -r 1 [ 1428.971952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e network_timeout -r 1 [ 1429.008451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e network_timeout -r 1 [ 1429.048579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e network_timeout -r 1 [ 1429.082636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e network_timeout -r 1 [ 1429.122168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e network_timeout -r 1 [ 1429.174965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1429.202318] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1429.480250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1429.546475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1429.600189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1429.642124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1429.676796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1429.710659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1429.743868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1429.777557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1429.798784] LNet: 67553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1429.819940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.851780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1429.916417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.963479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.038646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1430.075416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1430.111460] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1430.135394] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1432.160179] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1436.243206] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1436.750719] LNet: 67631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.755356] LNetError: 67631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.762936] LNet: Removed LNI 192.168.204.58@tcp [ 1436.780870] LNet: Removed LNI 192.168.204.58@tcp1 [ 1437.227608] Key type .llcrypt unregistered [ 1437.231535] Key type ._llcrypt unregistered [ 1454.863670] Key type ._llcrypt registered [ 1454.866355] Key type .llcrypt registered [ 1456.337280] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:25:21 (1787707521) [ 1456.929062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.947213] alg: No test for adler32 (adler32-zlib) [ 1457.936373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.990757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1458.013769] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1458.833113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1458.868670] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1459.791254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 1459.872948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1459.932926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1460.004273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1460.069676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1460.141934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1460.198924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.58@tcp [ 1460.227886] LNetError: 68732:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.58@tcp to recovery [ 1461.023297] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1461.023445] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1463.071610] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1463.894064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1467.169047] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1467.510601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1467.550208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1467.639838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.678284] LNet: 68839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.689896] LNetError: 68839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.700683] LNet: Removed LNI 192.168.204.58@tcp [ 1467.724670] LNet: Removed LNI 192.168.204.58@tcp1 [ 1467.755498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.845456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.879773] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1468.798486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1468.830667] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1469.734671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 1469.776352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.858350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1469.911602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1469.993671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1470.038931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1470.085867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1470.127699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.58@tcp [ 1470.158105] LNetError: 69112:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.58@tcp to recovery [ 1470.879621] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1473.691935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1477.024637] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1477.032465] LNet: Skipped 1 previous similar message [ 1481.937419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1481.967519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1482.011920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1490.158675] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:25:55 (1787707555) [ 1490.289474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.342257] LNet: 69822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.352908] LNetError: 69822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.360883] LNetError: 69822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1490.368483] LNet: Removed LNI 192.168.204.58@tcp [ 1490.378930] LNet: Removed LNI 192.168.204.58@tcp1 [ 1490.412380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.561379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.610782] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1491.629965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.653303] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1492.730389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 1492.884849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1492.944833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1492.998308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1493.044205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1493.075994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1493.176289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.58@tcp --health 0 [ 1493.396236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1493.599258] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.58@tcp [ 1494.625441] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1494.646729] LNet: Skipped 1 previous similar message [ 1509.043204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1509.173405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1509.206467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1509.237480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.58@tcp --health 500 [ 1509.369542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.155345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.220839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.255294] LNet: 70330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.263968] LNetError: 70330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.273939] LNetError: 70330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.279916] LNet: Removed LNI 192.168.204.58@tcp [ 1517.298171] LNet: Removed LNI 192.168.204.58@tcp1 [ 1517.322825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.397393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.431571] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1518.227269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.261188] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1519.145765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 1519.195476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.279527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1519.306897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1519.367716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1519.443411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1519.513430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1519.586949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.58@tcp --health 0 [ 1520.415420] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1520.428491] LNet: Skipped 3 previous similar messages [ 1520.940560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.319094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.354475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1531.394442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1539.630378] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:26:44 (1787707604) [ 1540.235281] LNet: 71376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.248705] LNetError: 71376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.259439] LNetError: 71376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1540.269043] LNet: Removed LNI 192.168.204.58@tcp [ 1540.298656] LNet: Removed LNI 192.168.204.58@tcp1 [ 1541.147390] Key type .llcrypt unregistered [ 1541.151442] Key type ._llcrypt unregistered [ 1547.785275] Key type ._llcrypt registered [ 1547.788691] Key type .llcrypt registered [ 1547.973850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1547.994618] alg: No test for adler32 (adler32-zlib) [ 1548.942579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1548.965960] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1558.389389] Lustre: DEBUG MARKER: Initial discovery [ 1558.472315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.158@tcp [ 1560.694642] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1560.978869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1561.040687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.158@tcp [ 1561.067207] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1561.103495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1561.150846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.158@tcp -r 1 -m GET -l 3 [ 1561.202254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.158@tcp -r 1 -m GET -e local_error [ 1561.248443] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1561.269751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.158@tcp -r 1 -m PUT -l 6 [ 1562.752426] Lustre: DEBUG MARKER: Force oleg458-client.virtnet to discover 192.168.204.158@tcp (in background) [ 1562.968601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.158@tcp [ 1564.563691] Lustre: DEBUG MARKER: Force oleg458-server to discover 192.168.204.58@tcp [ 1566.048273] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1567.014327] Lustre: DEBUG MARKER: Wait for 72143 [ 1568.722629] Lustre: DEBUG MARKER: Finished wait on 72143 [ 1569.166220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1569.229284] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1569.252166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1569.613791] LNet: 72597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.633562] LNetError: 72597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.654604] LNet: Removed LNI 192.168.204.58@tcp [ 1570.066553] Key type .llcrypt unregistered [ 1570.070426] Key type ._llcrypt unregistered [ 1587.458498] Key type ._llcrypt registered [ 1587.463790] Key type .llcrypt registered [ 1589.085937] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:27:34 (1787707654) [ 1589.676432] Key type .llcrypt unregistered [ 1589.685602] Key type ._llcrypt unregistered [ 1596.972730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1597.273222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1597.280825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1597.283230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1599.973957] Key type ._llcrypt registered [ 1599.976406] Key type .llcrypt registered [ 1600.162203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.181496] alg: No test for adler32 (adler32-zlib) [ 1601.204354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.287697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.316179] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1602.210243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1602.242766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1612.146977] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:27:57 (1787707677) [ 1612.567347] LNet: 74682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.572960] LNetError: 74682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1612.578876] LNet: Removed LNI 10.1.2.3@tcp [ 1612.594583] LNet: Removed LNI 192.168.204.58@tcp [ 1613.393997] Key type .llcrypt unregistered [ 1613.398989] Key type ._llcrypt unregistered [ 1620.865143] Key type ._llcrypt registered [ 1620.869952] Key type .llcrypt registered [ 1621.115428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1621.141606] alg: No test for adler32 (adler32-zlib) [ 1622.104941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.180542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.206535] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1622.760407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1622.779846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1624.055794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.58@tcp [ 1625.119679] LNetError: Unexpected error -22 connecting to 192.168.204.58@tcp at host 192.168.204.58:988 [ 1631.270943] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:28:17 (1787707697) [ 1631.542777] LNet: 75819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.552802] LNetError: 75819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1631.567243] LNet: Removed LNI 10.1.2.3@tcp [ 1631.584294] LNet: Removed LNI 192.168.204.58@tcp [ 1632.299764] Key type .llcrypt unregistered [ 1632.303293] Key type ._llcrypt unregistered [ 1639.124877] Key type ._llcrypt registered [ 1639.128153] Key type .llcrypt registered [ 1639.348335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.377148] alg: No test for adler32 (adler32-zlib) [ 1640.342436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.382882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.403314] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1641.276510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1641.301520] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 1642.390272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.58@tcp1 --nid 192.168.204.58@tcp2 [ 1647.541677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp1 192.168.204.58@tcp2 [ 1647.566287] LNetError: 76659:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.58@tcp1 to 192.168.204.58@tcp2 - Route table may be misconfigured [ 1647.593727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp2 192.168.204.58@tcp1 [ 1654.507801] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:28:40 (1787707720) [ 1654.764298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.813613] LNet: 77234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.828232] LNetError: 77234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.835711] LNet: Removed LNI 192.168.204.58@tcp1 [ 1655.912363] LNet: Removed LNI 192.168.204.58@tcp2 [ 1655.943797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.047525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.084456] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1657.070602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1657.113099] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1658.254048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 1658.297496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.58@tcp [ 1658.355033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp1 192.168.204.58@tcp1 [ 1658.410721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.58@tcp -d 192.168.204.58@tcp -e network_timeout [ 1658.475246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -e network_timeout [ 1658.524519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -e network_timeout [ 1658.569628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -e network_timeout [ 1658.625606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1658.651634] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1658.686612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1658.844296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1659.044578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1659.109859] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1666.876451] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:28:52 (1787707732) [ 1667.133388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.164686] LNet: 78096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.173374] LNetError: 78096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.184361] LNetError: 78096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1667.191049] LNet: Removed LNI 192.168.204.58@tcp [ 1668.279221] LNet: Removed LNI 192.168.204.58@tcp1 [ 1668.391127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.586502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1668.914694] LNet: 78137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.321420] Key type .llcrypt unregistered [ 1669.326780] Key type ._llcrypt unregistered [ 1681.465701] Key type ._llcrypt registered [ 1681.467660] Key type .llcrypt registered [ 1682.919214] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:29:08 (1787707748) [ 1683.614415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.646392] alg: No test for adler32 (adler32-zlib) [ 1684.575678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.174822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1687.987661] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:29:13 (1787707753) [ 1688.231693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.264916] LNet: 79237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.297777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.405727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.447751] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1689.324805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1689.354339] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1690.415726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1690.471852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp1 [ 1690.504689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp1 [ 1697.549455] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:29:23 (1787707763) [ 1699.607038] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg458-server [ 1701.533847] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:29:27 (1787707767) [ 1703.586362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg458-server [ 1705.746145] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:29:31 (1787707771) [ 1708.072623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg458-server [ 1710.390913] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:29:36 (1787707776) [ 1713.105491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg458-server [ 1715.279590] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:29:40 (1787707780) [ 1717.565139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg458-server [ 1719.559673] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:29:45 (1787707785) [ 1722.027953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg458-server [ 1724.395481] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:29:49 (1787707789) [ 1726.833100] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg458-server [ 1729.129723] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:29:54 (1787707794) [ 1731.622279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg458-server [ 1734.454684] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:29:59 (1787707799) [ 1737.620251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg458-server [ 1740.176489] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:30:05 (1787707805) [ 1740.296161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.333132] LNet: 83113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.352974] LNetError: 83113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.368779] LNet: Removed LNI 192.168.204.58@tcp [ 1741.418183] LNet: Removed LNI 192.168.204.58@tcp1 [ 1741.496535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.619176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.662311] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1742.719983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1742.865178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1743.113789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.135263] LNet: 83266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.141862] LNetError: 83266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.154989] LNetError: 83266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1743.176396] LNet: Removed LNI 192.168.204.58@tcp [ 1743.231219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.329279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1743.361395] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1744.220968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1744.477848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.522941] LNetError: 83411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.558917] LNet: Removed LNI 192.168.204.58@tcp [ 1744.612056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.699295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.733505] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1745.587271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1745.720375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1746.042443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.072544] LNet: 83559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.078477] LNet: 83559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1747.111844] LNet: Removed LNI 192.168.204.58@tcp [ 1747.191761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.332021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1747.401594] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1748.451977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1749.966278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.985950] LNetError: 83713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.991116] LNetError: 83713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1750.002961] LNet: Removed LNI 192.168.204.58@tcp [ 1750.024603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.093224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.133055] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1750.975669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1751.078163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1752.470460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.516426] LNet: 83870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.525253] LNet: 83870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1753.574540] LNet: Removed LNI 192.168.204.58@tcp [ 1753.616927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.724476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1753.747476] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1754.542505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1754.760075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.786088] LNetError: 84015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.791632] LNetError: 84015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1755.820254] LNet: Removed LNI 192.168.204.58@tcp [ 1755.866755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.995780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.027850] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1757.020929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1757.128741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1757.555252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.636663] LNet: Removed LNI 192.168.204.58@tcp [ 1758.675404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.766706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1758.818992] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1759.707867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1759.942353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.025330] LNet: Removed LNI 192.168.204.58@tcp [ 1761.105575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.239096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.270934] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1762.139936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1762.248657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1762.609894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.666560] LNet: 84456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.678539] LNet: 84456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1763.749976] LNet: Removed LNI 192.168.204.58@tcp [ 1763.804084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.893350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1763.918090] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1764.864890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1765.180710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.198637] LNetError: 84601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.202857] LNetError: 84601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1765.234557] LNet: Removed LNI 192.168.204.58@tcp [ 1765.270799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.360640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.401301] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1766.448982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1766.521437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1766.813254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.915821] LNet: Removed LNI 192.168.204.58@tcp [ 1767.976660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.073865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1768.112799] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1768.872613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 1769.157816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.213507] LNet: Removed LNI 192.168.204.58@tcp [ 1770.265177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.367990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.404560] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1771.353776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1771.419970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.473789] LNet: Removed LNI 192.168.204.58@tcp [ 1771.497992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.577149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1771.619322] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1772.447818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1772.579335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.621937] LNet: Removed LNI 192.168.204.58@tcp [ 1772.650666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.735235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1772.892769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.940302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.041429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1773.137424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.174619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.234125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1773.310541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.349719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.394260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1773.420425] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1774.292576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.345466] LNet: Removed LNI 192.168.204.58@tcp [ 1774.369792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.473138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1782.395857] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:30:48 (1787707848) [ 1782.493743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.518135] LNet: 85910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.527823] LNet: 85910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1782.568561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.691674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1782.722902] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/180] [ 1783.910182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.938469] LNetError: 86054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.943331] LNetError: 86054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1783.966600] LNet: Removed LNI 192.168.204.58@tcp231 [ 1783.991577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.064602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1784.088716] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/180] [ 1784.092795] LNet: 86062:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.719722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1784.763631] LNet: Removed LNI 192.168.204.58@tcp231 [ 1784.784165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1784.829259] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/180] [ 1786.241789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.289146] LNet: Removed LNI 192.168.204.58@tcp231 [ 1786.325693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.422787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1786.458041] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/0] [ 1786.470733] LNet: 86214:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1787.099461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1787.136511] LNet: Removed LNI 192.168.204.58@tcp231 [ 1787.153813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1787.179287] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/0] [ 1788.826949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.858587] LNet: Removed LNI 192.168.204.58@tcp231 [ 1788.890954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.008054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1789.062359] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/60] [ 1789.079357] LNet: 86366:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.720835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1789.770385] LNet: Removed LNI 192.168.204.58@tcp231 [ 1789.801285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1789.822502] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/60] [ 1791.046419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.083855] LNet: Removed LNI 192.168.204.58@tcp231 [ 1791.111037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.175910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1791.212795] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/180] [ 1791.218380] LNet: 86517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1791.668207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1791.703166] LNet: Removed LNI 192.168.204.58@tcp231 [ 1791.737574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1791.767042] LNet: Added LNI 192.168.204.58@tcp231 [8/256/0/180] [ 1801.459929] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:31:06 (1787707866) [ 1801.585327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.630518] LNet: Removed LNI 192.168.204.58@tcp231 [ 1801.673817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.808445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1801.829348] LNet: Added LNI 192.168.204.58@tcp232 [8/256/0/180] [ 1802.535810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1802.566872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1802.596946] LNet: Removed LNI 192.168.204.58@tcp232 [ 1802.626120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1802.675405] LNet: Added LNI 192.168.204.58@tcp232 [8/256/0/180] [ 1803.491916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1803.541822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1803.577227] LNet: Removed LNI 192.168.204.58@tcp232 [ 1803.589685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1803.621702] LNet: Added LNI 192.168.204.58@tcp232 [8/256/0/180] [ 1804.540213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1804.569854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1804.600710] LNet: Removed LNI 192.168.204.58@tcp232 [ 1804.625620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1804.662817] LNet: Added LNI 192.168.204.58@tcp232 [8/256/0/180] [ 1805.503072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1805.553199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1805.606172] LNet: Removed LNI 192.168.204.58@tcp232 [ 1805.635614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1813.343943] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:31:18 (1787707878) [ 1815.061461] Key type .llcrypt unregistered [ 1815.069861] Key type ._llcrypt unregistered [ 1821.575688] Key type ._llcrypt registered [ 1821.582868] Key type .llcrypt registered [ 1821.924959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1821.955933] alg: No test for adler32 (adler32-zlib) [ 1822.982588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1823.017996] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1832.004854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1843.748829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.808761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1843.866105] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1846.286958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_interrupt -r 2 [ 1846.345425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_interrupt -r 2 [ 1846.423132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_interrupt -r 2 [ 1846.555190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_interrupt -r 2 [ 1846.640668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_interrupt -r 2 [ 1846.687221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_interrupt -r 2 [ 1846.746952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_interrupt -r 2 [ 1846.817491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_interrupt -r 2 [ 1846.919957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1846.946067] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1846.961241] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1847.193710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1847.224451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1847.292972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1847.325357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1847.357349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1847.391551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1847.428404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1847.463710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1847.489681] LNet: 89026:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1847.512096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1847.558762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1847.617663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1847.698297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1847.765346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1847.793881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1848.445261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_dropped -r 2 [ 1848.500794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_dropped -r 2 [ 1848.536241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_dropped -r 2 [ 1848.586094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_dropped -r 2 [ 1848.625926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_dropped -r 2 [ 1848.679931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_dropped -r 2 [ 1848.713440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_dropped -r 2 [ 1848.752813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_dropped -r 2 [ 1848.798088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1848.829673] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1849.079541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1849.138816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1849.167478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1849.216942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1849.268610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1849.361967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1849.424904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1849.465544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1849.493903] LNet: 89147:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1849.519412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1849.575522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1849.626154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1849.678599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1849.727232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1849.769313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1850.412869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_aborted -r 2 [ 1850.452818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_aborted -r 2 [ 1850.506964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_aborted -r 2 [ 1850.564873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_aborted -r 2 [ 1850.624588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_aborted -r 2 [ 1850.687185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_aborted -r 2 [ 1850.731540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_aborted -r 2 [ 1850.779776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_aborted -r 2 [ 1850.844617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1850.877836] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1851.188866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1851.220077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1851.258784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1851.306941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1851.366580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1851.437024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1851.495783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1851.547896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1851.572232] LNet: 89267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.593334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.623619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.656624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.703203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.745866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.787865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1852.412905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_no_route -r 2 [ 1852.443650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_no_route -r 2 [ 1852.481796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_no_route -r 2 [ 1852.542896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_no_route -r 2 [ 1852.597613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_no_route -r 2 [ 1852.682189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_no_route -r 2 [ 1852.773205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_no_route -r 2 [ 1852.853198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_no_route -r 2 [ 1852.922738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1852.961649] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1853.250457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1853.304367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1853.348655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1853.389610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1853.431757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1853.474839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1853.527884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1853.571395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1853.598762] LNet: 89387:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1853.628026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1853.678132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1853.714811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1853.776611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1853.826409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1853.877642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1854.569655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e local_timeout -r 2 [ 1854.615854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e local_timeout -r 2 [ 1854.656885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e local_timeout -r 2 [ 1854.711231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e local_timeout -r 2 [ 1854.789587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e local_timeout -r 2 [ 1854.865710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e local_timeout -r 2 [ 1854.951224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e local_timeout -r 2 [ 1855.048185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e local_timeout -r 2 [ 1855.131266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1855.539512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1855.591469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1855.642679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1855.679774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1855.711491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1855.739512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1855.773858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1855.813534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1855.861721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1855.908557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1855.971298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1856.029304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1856.082261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1856.127789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1857.025817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp -m GET -e remote_dropped -r 2 [ 1857.076582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp -d 192.168.204.158@tcp1 -m GET -e remote_dropped -r 2 [ 1857.149827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp -m GET -e remote_dropped -r 2 [ 1857.217716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp -d 192.168.204.58@tcp1 -m GET -e remote_dropped -r 2 [ 1857.288808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp -m GET -e remote_dropped -r 2 [ 1857.361996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.58@tcp1 -d 192.168.204.158@tcp1 -m GET -e remote_dropped -r 2 [ 1857.398866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp -m GET -e remote_dropped -r 2 [ 1857.443637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.58@tcp1 -d 192.168.204.58@tcp1 -m GET -e remote_dropped -r 2 [ 1857.493392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.58@tcp 192.168.204.158@tcp [ 1857.532534] LNet: There was an unexpected network error while writing to 192.168.204.158: rc = -22 [ 1857.538514] LNet: Skipped 1 previous similar message [ 1857.544553] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.158@tcp [ 1857.915616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.986706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1858.030034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.081987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.124468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.216474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1858.281488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1858.316546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1858.358868] LNet: 89627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1858.367202] LNet: 89627:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1858.404899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1858.470304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1858.563714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1858.618742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1858.670855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.753132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.525275] LNet: 89706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.534828] LNetError: 89706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.543084] LNet: Removed LNI 192.168.204.58@tcp [ 1866.558723] LNet: Removed LNI 192.168.204.58@tcp1 [ 1867.327090] Key type .llcrypt unregistered [ 1867.331692] Key type ._llcrypt unregistered [ 1888.738075] Key type ._llcrypt registered [ 1888.739393] Key type .llcrypt registered [ 1890.185777] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:32:35 (1787707955) [ 1890.784505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1890.795424] alg: No test for adler32 (adler32-zlib) [ 1891.829028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.908954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.955403] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1893.036970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1893.083461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.58@tcp [ 1893.114735] LNetError: 90685:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.58@tcp to recovery [ 1893.222413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.58@tcp [ 1893.349923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1901.780988] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:32:47 (1787707967) [ 1902.137310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.163037] LNet: 91275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.179672] LNetError: 91275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.220203] LNet: Removed LNI 192.168.204.58@tcp [ 1902.257495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.348700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.397580] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1903.214590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1903.254449] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 1904.292887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.58@tcp [ 1904.324972] LNetError: 91541:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.58@tcp to recovery [ 1904.357181] LNet: 1 local NIs in recovery (showing 1): 192.168.204.58@tcp [ 1904.366289] LNet: There was an unexpected network error while writing to 192.168.204.58: rc = -22 [ 1904.502257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.58@tcp [ 1914.203167] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:32:59 (1787707979) [ 1916.408569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg458-server [ 1918.210322] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:33:04 (1787707984) [ 1920.478511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg458-server [ 1922.810696] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:33:08 (1787707988) [ 1924.897259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg458-server [ 1927.115107] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:33:12 (1787707992) [ 1927.449143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.502567] LNet: 93163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.519655] LNetError: 93163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.547570] LNet: Removed LNI 192.168.204.58@tcp [ 1928.551579] LNetError: 93163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.584056] LNet: Removed LNI 192.168.204.58@tcp1 [ 1928.644588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.724456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.764892] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1929.634602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1929.731801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1929.796692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1929.886297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1929.924612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1930.011941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1937.195810] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:33:22 (1787708002) [ 1937.537259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.571229] LNet: 93894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.578570] LNetError: 93894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.594604] LNet: Removed LNI 192.168.204.58@tcp [ 1937.627672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.713421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.745945] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1947.141471] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:33:32 (1787708012) [ 1947.695199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.754335] LNet: 94598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.763280] LNetError: 94598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.778794] LNet: Removed LNI 192.168.204.58@tcp [ 1947.813161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.878985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.912797] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1956.480837] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:33:41 (1787708021) [ 1958.378580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1960.704749] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:33:46 (1787708026) [ 1961.412615] LNet: 95633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.421284] LNetError: 95633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.436508] LNet: Removed LNI 192.168.204.58@tcp [ 1962.271885] Key type .llcrypt unregistered [ 1962.277972] Key type ._llcrypt unregistered [ 1971.155479] Key type ._llcrypt registered [ 1971.157048] Key type .llcrypt registered [ 1971.375426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1971.398585] alg: No test for adler32 (adler32-zlib) [ 1972.499329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1972.545704] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 1981.932734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 1987.984573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.158@tcp [ 1988.691508] LNet: 96059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.706137] LNetError: 96059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.747229] LNet: Removed LNI 192.168.204.58@tcp [ 1989.263572] Key type .llcrypt unregistered [ 1989.270879] Key type ._llcrypt unregistered [ 2003.731884] Key type ._llcrypt registered [ 2003.734819] Key type .llcrypt registered [ 2005.475371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2007.360145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2009.670694] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:34:35 (1787708075) [ 2011.585014] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2013.650303] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:34:39 (1787708079) [ 2016.449558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg458-server [ 2018.986928] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:34:44 (1787708084) [ 2021.321145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg458-server [ 2023.672630] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:34:49 (1787708089) [ 2026.086051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.112290] alg: No test for adler32 (adler32-zlib) [ 2027.308894] Lustre: Lustre: Build Version: 2.17.57_80_gcb5c033 [ 2027.663141] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2029.432836] Key type lgssc registered [ 2031.098055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.344647] Key type lgssc unregistered [ 2053.657523] LNet: 99243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.669805] LNetError: 99243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.685147] LNet: Removed LNI 192.168.204.58@tcp [ 2054.415166] Key type .llcrypt unregistered [ 2054.417404] Key type ._llcrypt unregistered [ 2072.052459] Key type ._llcrypt registered [ 2072.057793] Key type .llcrypt registered [ 2073.784803] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:35:39 (1787708139) [ 2074.186136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.204041] alg: No test for adler32 (adler32-zlib) [ 2075.196788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.256629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2075.294440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2075.346104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2075.416966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2082.958379] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:35:48 (1787708148) [ 2083.450972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.477093] LNet: 100661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.509669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.572757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.610689] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2084.730479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.750427] LNet: 100805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.755657] LNetError: 100805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.771647] LNet: Removed LNI 192.168.204.58@tcp [ 2084.790958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.840585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2084.861928] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2085.454723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.479881] LNetError: 100834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.493512] LNet: Removed LNI 192.168.204.58@tcp [ 2085.510468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.591155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2085.624586] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2094.525393] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:35:59 (1787708159) [ 2094.668273] LNet: 101417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.680825] LNet: 101417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2094.689315] LNetError: 101417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.743729] LNet: Removed LNI 192.168.204.58@tcp [ 2095.543644] Key type .llcrypt unregistered [ 2095.549316] Key type ._llcrypt unregistered [ 2103.865287] Key type ._llcrypt registered [ 2103.869230] Key type .llcrypt registered [ 2104.087791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2104.103525] alg: No test for adler32 (adler32-zlib) [ 2105.082919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.202565] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2105.214510] LNet: *** cfs_fail_loc=e003, val=100*** [ 2106.158516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2106.180324] LNet: *** cfs_fail_loc=e003, val=100*** [ 2106.185218] LNet: Skipped 2 previous similar messages [ 2106.193439] Ping reply received with mapping: [ 2106.196184] found metadata reply NI[192.168.204.58@tcp] [ 2106.199334] 3a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2106.202722] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2106.206100] 5a 5a 5a 5a 5a 5a 5a 5a [ 2106.460827] LNet: 101827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.474727] LNetError: 101827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.505172] LNet: Removed LNI 192.168.204.58@tcp [ 2107.414528] Key type .llcrypt unregistered [ 2107.418336] Key type ._llcrypt unregistered [ 2121.385307] Key type ._llcrypt registered [ 2121.390326] Key type .llcrypt registered [ 2122.943835] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:36:28 (1787708188) [ 2123.950679] Key type .llcrypt unregistered [ 2123.955083] Key type ._llcrypt unregistered [ 2131.604598] Key type ._llcrypt registered [ 2131.611311] Key type .llcrypt registered [ 2131.876676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.902732] alg: No test for adler32 (adler32-zlib) [ 2133.190242] LNetError: 102896:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2133.735428] Key type .llcrypt unregistered [ 2133.737737] Key type ._llcrypt unregistered [ 2139.506818] Key type ._llcrypt registered [ 2139.510817] Key type .llcrypt registered [ 2141.028516] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:36:46 (1787708206) [ 2141.718395] Key type .llcrypt unregistered [ 2141.720587] Key type ._llcrypt unregistered [ 2148.630705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2148.640695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2150.237743] Key type ._llcrypt registered [ 2150.244829] Key type .llcrypt registered [ 2150.543997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2150.570663] alg: No test for adler32 (adler32-zlib) [ 2151.663977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.775625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.851599] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:37:08 (1787708228) [ 2163.132782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.184579] LNet: 104681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.201808] LNetError: 104681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2163.224279] LNet: Removed LNI 10.1.2.3@tcp [ 2163.260506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.450491] Key type .llcrypt unregistered [ 2164.453233] Key type ._llcrypt unregistered [ 2171.738885] Key type ._llcrypt registered [ 2171.742783] Key type .llcrypt registered [ 2172.086913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2172.119500] alg: No test for adler32 (adler32-zlib) [ 2173.087819] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2173.185902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2174.547567] LNet: 104982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.575811] LNetError: 104982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.594532] LNet: Removed LNI 192.168.204.58@tcp [ 2175.463681] Key type .llcrypt unregistered [ 2175.465735] Key type ._llcrypt unregistered [ 2187.814752] Key type ._llcrypt registered [ 2187.817712] Key type .llcrypt registered [ 2189.277309] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:37:34 (1787708254) [ 2190.299165] Key type .llcrypt unregistered [ 2190.301160] Key type ._llcrypt unregistered [ 2198.089110] Key type ._llcrypt registered [ 2198.091296] Key type .llcrypt registered [ 2198.350764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2198.378594] alg: No test for adler32 (adler32-zlib) [ 2205.722296] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:37:51 (1787708271) [ 2206.152819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.225252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.375827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.426819] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2207.392985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.432708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2207.465156] LNet: Removed LNI 192.168.204.58@tcp [ 2207.754577] LNet: 106740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.231390] Key type .llcrypt unregistered [ 2208.239480] Key type ._llcrypt unregistered [ 2222.130636] Key type ._llcrypt registered [ 2222.139361] Key type .llcrypt registered [ 2223.789099] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:38:09 (1787708289) [ 2224.230483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.244831] alg: No test for adler32 (adler32-zlib) [ 2225.255470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.335485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.359768] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2226.118438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2226.154873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.58@tcp [ 2226.173928] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2226.184286] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2226.189305] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2226, 0, 0 [ 2226.196557] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2226.200287] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2226.209356] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2226.215469] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2226, 0, 0 [ 2226.225234] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2226.230631] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2226.235614] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2226.240462] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2226, 0, 2276 [ 2226.248212] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2226.253250] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2226.257994] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2226.260983] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2226, 0, 2276 [ 2226.268112] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2233.127756] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:38:19 (1787708299) [ 2233.681794] LNet: 108280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.696249] LNetError: 108280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.713885] LNet: Removed LNI 192.168.204.58@tcp [ 2234.301599] Key type .llcrypt unregistered [ 2234.303866] Key type ._llcrypt unregistered [ 2241.286636] Key type ._llcrypt registered [ 2241.289854] Key type .llcrypt registered [ 2241.556760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2241.576481] alg: No test for adler32 (adler32-zlib) [ 2242.584055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2242.607728] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2249.455695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 2258.302763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.361841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2258.398770] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 2259.486403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2259.531878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2259.536717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2259.541052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2261.918415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2261.952708] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2271.660605] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:38:57 (1787708337) [ 2271.782465] LNet: 109785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.789708] LNetError: 109785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.809639] LNet: Removed LNI 192.168.204.58@tcp [ 2272.864509] LNet: Removed LNI 192.168.204.58@tcp1 [ 2272.869297] LNetError: 109785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2272.878420] LNetError: 109785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2272.893775] LNet: Removed LNI 10.1.2.3@tcp99 [ 2273.528901] Key type .llcrypt unregistered [ 2273.531708] Key type ._llcrypt unregistered [ 2280.155615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2280.158944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2282.837389] Key type ._llcrypt registered [ 2282.839660] Key type .llcrypt registered [ 2283.084697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2283.134146] alg: No test for adler32 (adler32-zlib) [ 2283.998951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.042830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2284.062990] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2284.672795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2284.684649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2286.711984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2286.767801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2286.831140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2286.850390] LNetError: 110542:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2286.882542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2286.931093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2287.252867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2287.330664] LNet: 110562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.335964] LNetError: 110562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2287.340957] LNet: Removed LNI 10.1.2.3@tcp [ 2288.366947] LNet: Removed LNI 192.168.204.58@tcp [ 2289.228894] Key type .llcrypt unregistered [ 2289.230973] Key type ._llcrypt unregistered [ 2303.769904] Key type ._llcrypt registered [ 2303.772552] Key type .llcrypt registered [ 2305.719586] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:39:30 (1787708370) [ 2306.281973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.336791] alg: No test for adler32 (adler32-zlib) [ 2307.434445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.549851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.613935] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2309.001979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2309.144630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg458-client@tcp [ 2317.662905] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:39:42 (1787708382) [ 2317.822129] LNet: 112093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.835323] LNetError: 112093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.864538] LNet: Removed LNI 192.168.204.58@tcp [ 2318.486534] Key type .llcrypt unregistered [ 2318.489683] Key type ._llcrypt unregistered [ 2325.209444] Key type ._llcrypt registered [ 2325.213110] Key type .llcrypt registered [ 2325.380241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.405909] alg: No test for adler32 (adler32-zlib) [ 2326.341645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2326.385784] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2326.451389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2333.480176] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:39:59 (1787708399) [ 2335.114952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2336.935773] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:40:02 (1787708402) [ 2338.429302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2340.500514] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:40:06 (1787708406) [ 2340.761271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.780806] LNet: 113528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.790648] LNetError: 113528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.811589] LNet: Removed LNI 192.168.204.58@tcp [ 2340.841941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.888901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2340.912973] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2341.600279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2341.638792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2348.349403] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:40:14 (1787708414) [ 2348.713934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.738462] LNet: 114237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.753810] LNetError: 114237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.774056] LNet: Removed LNI 192.168.204.58@tcp [ 2348.789890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.868396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2348.912420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2349.588547] Key type .llcrypt unregistered [ 2349.590697] Key type ._llcrypt unregistered [ 2360.930553] Key type ._llcrypt registered [ 2360.932805] Key type .llcrypt registered [ 2362.155281] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:40:28 (1787708428) [ 2362.678921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.697512] alg: No test for adler32 (adler32-zlib) [ 2363.576854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.617625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2363.634387] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2364.294764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2364.358167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2370.713126] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:40:36 (1787708436) [ 2370.886484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.919883] LNet: 115792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2370.928596] LNetError: 115792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.964023] LNet: Removed LNI 192.168.204.58@tcp [ 2371.058239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.111517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2371.151068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2378.472117] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:40:44 (1787708444) [ 2378.720765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.748574] LNet: 116377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.789797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.934563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2379.306596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2379.344823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2379.390796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2379.746113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2379.780415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2379.883453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.908831] LNet: 116410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.934453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.990733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2380.462963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2380.510509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2380.566980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2381.052216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2381.115055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2388.792383] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:40:54 (1787708454) [ 2389.037344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.058290] LNet: 117011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.092582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.158879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2389.176333] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2390.222766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2390.254675] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2391.324424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2391.420893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2391.711750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.742947] LNetError: 117291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.799295] LNet: Removed LNI 192.168.204.58@tcp [ 2392.809137] LNetError: 117291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.882826] LNet: Removed LNI 192.168.204.58@tcp2 [ 2392.929091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.985041] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2392.992036] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2392.998468] LNet: 117297:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.276154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2393.424246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.468703] LNet: 117334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.471659] LNet: 117334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2394.527170] LNet: Removed LNI 192.168.204.58@tcp [ 2394.544576] LNet: Removed LNI 192.168.204.58@tcp2 [ 2394.573499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.645301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.667883] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2395.546271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2395.569977] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2396.762649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2396.792791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2396.844449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2397.131963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.157839] LNetError: 117619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.163556] LNetError: 117619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2398.175884] LNet: Removed LNI 192.168.204.58@tcp [ 2399.208365] LNet: Removed LNI 192.168.204.58@tcp2 [ 2399.241195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.317795] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2399.325682] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2399.330025] LNet: 117625:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2400.080466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2400.299342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.332485] LNet: Removed LNI 192.168.204.58@tcp [ 2401.381970] LNet: Removed LNI 192.168.204.58@tcp2 [ 2401.419370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.502611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2401.525705] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2402.200822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2402.224447] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2403.128468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2403.197750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2403.396995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.418454] LNet: 117942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.423418] LNet: 117942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2403.429325] LNetError: 117942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.434948] LNetError: 117942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2404.447259] LNet: Removed LNI 192.168.204.58@tcp [ 2404.460334] LNet: Removed LNI 192.168.204.58@tcp2 [ 2404.489640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.582343] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2404.590869] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2404.602626] LNet: 117948:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2405.470445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2405.699828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.752236] LNet: Removed LNI 192.168.204.58@tcp [ 2406.764654] LNet: Removed LNI 192.168.204.58@tcp2 [ 2406.819304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.876921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.910420] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2407.794413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2407.840862] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2408.678684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2408.710680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2408.751564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2408.954925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.020138] LNet: Removed LNI 192.168.204.58@tcp [ 2411.060750] LNet: Removed LNI 192.168.204.58@tcp2 [ 2411.131788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.213152] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2411.222348] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2411.225957] LNet: 118276:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2411.980394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2412.307484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.344761] LNetError: 118314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.350778] LNetError: 118314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2412.358197] LNet: Removed LNI 192.168.204.58@tcp [ 2413.419745] LNet: Removed LNI 192.168.204.58@tcp2 [ 2413.477420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.564959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.616324] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2414.390260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2414.417393] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2415.545604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2415.577503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2415.633361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2415.969706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.055910] LNet: Removed LNI 192.168.204.58@tcp [ 2417.070126] LNet: Removed LNI 192.168.204.58@tcp2 [ 2417.098903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.185467] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2417.196397] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2417.201658] LNet: 118617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2417.697262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2418.010864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2419.103167] LNet: Removed LNI 192.168.204.58@tcp [ 2419.121350] LNet: Removed LNI 192.168.204.58@tcp2 [ 2419.158305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.257168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2419.275784] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2420.157323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2420.178711] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2421.316895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2421.358362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2421.397276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2421.498503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2421.867898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.887797] LNet: 118971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.893336] LNet: 118971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2422.943233] LNet: Removed LNI 192.168.204.58@tcp [ 2423.973967] LNet: Removed LNI 192.168.204.58@tcp2 [ 2424.014731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.139340] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2424.155319] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2424.649864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2432.567407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2434.642240] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:41:40 (1787708500) [ 2434.707355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2434.722517] LNetError: 119730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.727405] LNetError: 119730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2434.732503] LNet: Removed LNI 192.168.204.58@tcp [ 2434.744843] LNet: Removed LNI 192.168.204.58@tcp2 [ 2434.774512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.161879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.189247] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2435.849901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2435.866121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2436.642736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.659160] LNet: Removed LNI 10.1.2.3@tcp [ 2436.670877] LNet: Removed LNI 192.168.204.58@tcp [ 2436.701870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.775323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2436.818421] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2436.825229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2437.361794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2437.399138] LNet: Removed LNI 192.168.204.58@tcp [ 2437.413144] LNet: Removed LNI 10.1.2.3@tcp [ 2444.402387] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:41:50 (1787708510) [ 2444.613192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2444.676944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.733585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2444.756245] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2445.652941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2445.722816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2445.765371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2445.802770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2445.844537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2445.913808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2445.997102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2446.040319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2446.089425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2446.129544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2446.175846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.58@tcp [ 2446.357523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2453.286335] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:41:58 (1787708518) [ 2453.535260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.632989] LNet: Removed LNI 192.168.204.58@tcp [ 2454.681952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.828676] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2455.006809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2455.015274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2455.026722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2457.311775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2457.357437] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2457.927586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2457.951627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2459.757968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2459.808407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2459.871767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.58@tcp -m PUT -e local_timeout -r 1 [ 2459.987563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2460.076844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2460.081723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2460.417281] LNetError: 121412:0:(lib-move.c:4801:lnet_parse()) 192.168.204.58@tcp, src 192.168.204.58@tcp: Dropping PUT (error -108 looking up sender) [ 2461.476234] LNet: 121874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.484854] LNet: 121874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2461.491931] LNet: Removed LNI 10.1.2.3@tcp [ 2461.508983] LNet: Removed LNI 192.168.204.58@tcp [ 2461.910271] Key type .llcrypt unregistered [ 2461.913732] Key type ._llcrypt unregistered [ 2474.627807] Key type ._llcrypt registered [ 2474.631528] Key type .llcrypt registered [ 2475.953920] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:42:21 (1787708541) [ 2476.758805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.778781] alg: No test for adler32 (adler32-zlib) [ 2477.751114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.162431] LNet: 122729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.695406] Key type .llcrypt unregistered [ 2478.699425] Key type ._llcrypt unregistered [ 2485.353984] Key type ._llcrypt registered [ 2485.357625] Key type .llcrypt registered [ 2485.552863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2485.572721] alg: No test for adler32 (adler32-zlib) [ 2486.480887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2486.527386] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2494.338946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 2494.538706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2495.141798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.158@tcp [ 2495.342597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2495.365359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2495.392838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2495.419418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2495.445454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2495.467656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2495.489364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2495.512557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2495.525582] LNet: 123199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2495.540398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2495.563190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2495.585375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2495.610573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2495.636356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2495.661754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2496.288619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2496.518095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2496.550936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2496.575989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2496.602813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2496.632819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2496.658936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2496.684181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2496.711773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2496.726888] LNet: 123316:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2496.741902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2496.763903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2496.793804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2496.800089] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.158@tcp [ 2496.839130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2496.877955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2496.914935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2497.498323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.158@tcp [ 2497.776862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2497.832968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2497.875107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2497.912725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2497.964738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2497.998690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.025357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.057291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.071306] LNet: 123433:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.096767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.141668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.179365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.217706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.254687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.305960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2499.058225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2499.320504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2499.377746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2499.425148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2499.477373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2499.522897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2499.556942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2499.594181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2499.628248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2499.681788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2499.733518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2499.806746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2499.891786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2499.934676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2499.969972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2500.367822] LNet: 123601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.382271] LNetError: 123601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.416319] LNet: Removed LNI 192.168.204.58@tcp [ 2501.849448] Key type .llcrypt unregistered [ 2501.853488] Key type ._llcrypt unregistered [ 2519.140792] Key type ._llcrypt registered [ 2519.144406] Key type .llcrypt registered [ 2520.699802] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:43:06 (1787708586) [ 2521.200740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.234151] alg: No test for adler32 (adler32-zlib) [ 2522.182942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.698879] LNet: 124490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.354590] Key type .llcrypt unregistered [ 2523.359604] Key type ._llcrypt unregistered [ 2529.761778] Key type ._llcrypt registered [ 2529.764479] Key type .llcrypt registered [ 2530.007671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2530.024275] alg: No test for adler32 (adler32-zlib) [ 2531.000512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2531.038919] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2538.485777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.158@tcp [ 2548.312117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2548.368175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2548.389735] LNet: Added LNI 192.168.204.58@tcp1 [8/256/0/180] [ 2549.266963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2549.856659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.158@tcp [ 2550.086397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2550.137560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2550.177728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2550.213976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2550.281822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2550.339276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2550.409768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2550.447735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2550.476800] LNet: 125147:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2550.495406] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.158@tcp [ 2550.502740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2550.532181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2550.559676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2550.590261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2550.630609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2550.653922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2551.275718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2551.553399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2551.593443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2551.650753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2551.694885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2551.742980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2551.771199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2551.808576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2551.845592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2551.875709] LNet: 125264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2551.896722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2551.940294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2551.970350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2552.008346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2552.056451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2552.101442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2552.761138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.158@tcp [ 2552.988852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2553.031622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2553.069852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2553.125042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2553.176791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2553.235180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2553.284327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2553.321702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2553.358495] LNet: 125381:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2553.373610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2553.394076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2553.425544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2553.472270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2553.501499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2553.531806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2554.155709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2554.415256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2554.444434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2554.476907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2554.532346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2554.566544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2554.601232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2554.651737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2554.689942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2554.732727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2554.765958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2554.811608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2554.851777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2554.915524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2554.971651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2560.772968] LNet: 125584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.784462] LNetError: 125584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.791705] LNet: Removed LNI 192.168.204.58@tcp [ 2560.817871] LNet: Removed LNI 192.168.204.58@tcp1 [ 2561.206204] Key type .llcrypt unregistered [ 2561.208885] Key type ._llcrypt unregistered [ 2579.684096] Key type ._llcrypt registered [ 2579.690868] Key type .llcrypt registered [ 2581.285795] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:44:06 (1787708646) [ 2581.786621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.800134] alg: No test for adler32 (adler32-zlib) [ 2582.678794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.749797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2582.787492] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2583.607324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2583.631451] LNet: Added LNI 192.168.204.58@tcp2 [8/256/0/180] [ 2584.652156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.58@tcp [ 2584.730654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2585.823540] LNet: Removed LNI 192.168.204.58@tcp [ 2592.421663] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:44:18 (1787708658) [ 2594.263308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2596.049419] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:44:21 (1787708661) [ 2604.049728] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:44:30 (1787708670) [ 2611.812553] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:44:37 (1787708677) [ 2620.153236] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:44:45 (1787708685) [ 2627.385860] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:44:53 (1787708693) [ 2636.933466] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:45:02 (1787708702) [ 2646.459491] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:45:11 (1787708711) [ 2655.816414] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:45:21 (1787708721) [ 2663.599253] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:45:29 (1787708729) [ 2671.520402] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:45:37 (1787708737) [ 2680.284511] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:45:45 (1787708745) [ 2688.320993] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:45:53 (1787708753) [ 2695.866271] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:46:01 (1787708761) [ 2703.756850] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:46:09 (1787708769) [ 2711.702221] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:46:17 (1787708777) [ 2718.384691] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:46:24 (1787708784) [ 2725.942509] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:46:31 (1787708791) [ 2733.534208] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:46:39 (1787708799) [ 2733.709746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2733.731426] LNet: 137280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.738830] LNetError: 137280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2733.750861] LNet: Removed LNI 192.168.204.58@tcp2 [ 2733.777063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.887209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2733.906885] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2734.837453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2734.854331] LNet: 137422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.862370] LNetError: 137422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.873281] LNet: Removed LNI 192.168.204.58@tcp [ 2734.908726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2734.966690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2734.992503] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2742.429050] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:46:48 (1787708808) [ 2751.549915] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:46:57 (1787708817) [ 2760.389598] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:47:06 (1787708826) [ 2769.179744] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:47:14 (1787708834) [ 2777.800973] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:47:23 (1787708843) [ 2786.124987] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:47:31 (1787708851) [ 2794.901743] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:47:40 (1787708860) [ 2803.145692] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:47:49 (1787708869) [ 2811.269851] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:47:57 (1787708877) [ 2820.284294] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:48:05 (1787708885) [ 2828.506600] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:48:14 (1787708894) [ 2836.829632] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:48:22 (1787708902) [ 2845.170780] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2592 sec ========== 21:48:31 (1787708911) [ 2845.809657] LNet: 144891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.817569] LNetError: 144891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2845.843489] LNet: Removed LNI 192.168.204.58@tcp [ 2846.596891] Key type .llcrypt unregistered [ 2846.605298] Key type ._llcrypt unregistered [ 2859.759760] Key type ._llcrypt registered [ 2859.765064] Key type .llcrypt registered [ 2860.499208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2860.535426] alg: No test for adler32 (adler32-zlib) [ 2861.678172] Lustre: Lustre: Build Version: 2.17.57_80_gcb5c033 [ 2862.092968] LNet: Added LNI 192.168.204.58@tcp [8/256/0/180] [ 2863.807187] Key type lgssc registered [ 2865.149509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2937.412997] Lustre: Mounted lustre-client - version 2.17.57_80_gcb5c033 [ 2942.808218] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2950.365880] Lustre: Unmounted lustre-client [ 2985.499127] Key type lgssc unregistered [ 2985.784957] LNet: 147852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2985.794280] LNetError: 147852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2985.834139] LNet: Removed LNI 192.168.204.58@tcp [ 2986.546251] Key type .llcrypt unregistered [ 2986.551599] Key type ._llcrypt unregistered