[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461167205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006889] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011026] SELinux: Initializing. [ 0.011552] *** VALIDATE selinux *** [ 0.018678] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022511] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025084] *** VALIDATE tmpfs *** [ 0.026348] *** VALIDATE proc *** [ 0.027184] *** VALIDATE cgroup *** [ 0.027629] *** VALIDATE cgroup2 *** [ 0.028202] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029108] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031022] Spectre V2 : User space: Vulnerable [ 0.032004] Speculative Store Bypass: Vulnerable [ 0.034289] debug: unmapping init [mem 0xffffffff9c059000-0xffffffff9c060fff] [ 0.036207] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037604] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038022] ... version: 2 [ 0.038831] ... bit width: 48 [ 0.039014] ... generic registers: 4 [ 0.040010] ... value mask: 0000ffffffffffff [ 0.041010] ... max period: 00007fffffffffff [ 0.042016] ... fixed-purpose events: 3 [ 0.042937] ... event mask: 000000070000000f [ 0.043322] rcu: Hierarchical SRCU implementation. [ 0.045427] smp: Bringing up secondary CPUs ... [ 0.046421] x86: Booting SMP configuration: [ 0.047016] .... node #0, CPUs: #1 #2 #3 [ 0.050129] smp: Brought up 1 node, 4 CPUs [ 0.051846] smpboot: Max logical packages: 1 [ 0.052012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123022] node 0 deferred pages initialised in 69ms [ 0.127142] devtmpfs: initialized [ 0.129269] x86/mm: Memory block size: 128MB [ 0.131964] gcov: version magic: 0x41383552 [ 0.133651] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141357] pinctrl core: initialized pinctrl subsystem [ 0.143165] [ 0.143594] ************************************************************* [ 0.145011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148016] ** ** [ 0.151015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153012] ** ** [ 0.156016] ** This means that this kernel is built to expose internal ** [ 0.159015] ** IOMMU data structures, which may compromise security on ** [ 0.162021] ** your system. ** [ 0.164011] ** ** [ 0.166026] ** If you see this message and you are not debugging the ** [ 0.169015] ** kernel, report this immediately to your vendor! ** [ 0.172015] ** ** [ 0.175013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177012] ************************************************************* [ 0.181186] NET: Registered protocol family 16 [ 0.182495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189090] cpuidle: using governor menu [ 0.190828] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193564] PCI: Using configuration type 1 for base access [ 0.196121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208042] cryptd: max_cpu_qlen set to 1000 [ 0.210199] ACPI: Added _OSI(Module Device) [ 0.212013] ACPI: Added _OSI(Processor Device) [ 0.213012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217251] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222373] ACPI: Interpreter enabled [ 0.223045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225013] ACPI: Using IOAPIC for interrupt routing [ 0.226087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240124] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254106] acpiphp: Slot [2] registered [ 0.255095] acpiphp: Slot [5] registered [ 0.256096] acpiphp: Slot [6] registered [ 0.257093] acpiphp: Slot [3] registered [ 0.258064] acpiphp: Slot [4] registered [ 0.259077] acpiphp: Slot [7] registered [ 0.260064] acpiphp: Slot [8] registered [ 0.261114] acpiphp: Slot [9] registered [ 0.262084] acpiphp: Slot [10] registered [ 0.263099] acpiphp: Slot [11] registered [ 0.264063] acpiphp: Slot [12] registered [ 0.265062] acpiphp: Slot [13] registered [ 0.266017] acpiphp: Slot [14] registered [ 0.266915] acpiphp: Slot [15] registered [ 0.267083] acpiphp: Slot [16] registered [ 0.269124] acpiphp: Slot [17] registered [ 0.271100] acpiphp: Slot [18] registered [ 0.272129] acpiphp: Slot [19] registered [ 0.274078] acpiphp: Slot [20] registered [ 0.276109] acpiphp: Slot [21] registered [ 0.277128] acpiphp: Slot [22] registered [ 0.279157] acpiphp: Slot [23] registered [ 0.281117] acpiphp: Slot [24] registered [ 0.282102] acpiphp: Slot [25] registered [ 0.283123] acpiphp: Slot [26] registered [ 0.285125] acpiphp: Slot [27] registered [ 0.286099] acpiphp: Slot [28] registered [ 0.288141] acpiphp: Slot [29] registered [ 0.289084] acpiphp: Slot [30] registered [ 0.291088] acpiphp: Slot [31] registered [ 0.292080] PCI host bridge to bus 0000:00 [ 0.293022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.296077] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309852] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312094] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317971] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.322055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.328015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.329460] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331481] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334526] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337926] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.346000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.351014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355997] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393027] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.438142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.440449] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.443374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.445463] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.448222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.454126] iommu: Default domain type: Passthrough [ 0.456455] SCSI subsystem initialized [ 0.458154] ACPI: bus type USB registered [ 0.460105] usbcore: registered new interface driver usbfs [ 0.461080] usbcore: registered new interface driver hub [ 0.463087] usbcore: registered new device driver usb [ 0.464107] pps_core: LinuxPPS API ver. 1 registered [ 0.465007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.467073] PTP clock support registered [ 0.468129] EDAC MC: Ver: 3.0.0 [ 0.470099] PCI: Using ACPI for IRQ routing [ 0.472464] NetLabel: Initializing [ 0.473009] NetLabel: domain hash size = 128 [ 0.475014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.477090] NetLabel: unlabeled traffic allowed by default [ 0.479042] vgaarb: loaded [ 0.480213] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.481007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.485597] clocksource: Switched to clocksource kvm-clock [ 0.587165] VFS: Disk quotas dquot_6.6.0 [ 0.588668] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.590888] *** VALIDATE ramfs *** [ 0.592298] *** VALIDATE hugetlbfs *** [ 0.594650] pnp: PnP ACPI init [ 0.596941] pnp: PnP ACPI: found 6 devices [ 0.612188] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.615487] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.617635] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.619702] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.622055] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.624325] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.627223] NET: Registered protocol family 2 [ 0.629722] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.634385] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.637791] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.642964] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.645717] TCP: Hash tables configured (established 65536 bind 65536) [ 0.647887] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.650126] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.652068] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.654423] NET: Registered protocol family 1 [ 0.656081] RPC: Registered named UNIX socket transport module. [ 0.658109] RPC: Registered udp transport module. [ 0.659880] RPC: Registered tcp transport module. [ 0.661138] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.663305] NET: Registered protocol family 44 [ 0.664848] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.666704] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.668637] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.670546] PCI: CLS 0 bytes, default 64 [ 0.671922] Unpacking initramfs... [ 1.930072] debug: unmapping init [mem 0xffff9e8e3cc64000-0xffff9e8e3ffcffff] [ 1.932561] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.933966] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.935628] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.439304] Initialise system trusted keyrings [ 2.441161] Key type blacklist registered [ 2.443835] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.449947] zbud: loaded [ 2.452543] *** VALIDATE nfs *** [ 2.453278] *** VALIDATE nfs4 *** [ 2.454398] pstore: using deflate compression [ 2.457086] Platform Keyring initialized [ 2.567376] NET: Registered protocol family 38 [ 2.568630] Key type asymmetric registered [ 2.569675] Asymmetric key parser 'x509' registered [ 2.571463] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.573416] io scheduler mq-deadline registered [ 2.574559] io scheduler kyber registered [ 2.575716] io scheduler bfq registered [ 2.577123] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.578870] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.581162] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.583657] ACPI: Power Button [PWRF] [ 2.588426] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.596625] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.608138] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.636697] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.663620] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.672399] Non-volatile memory driver v1.3 [ 2.673512] Linux agpgart interface v0.103 [ 2.705959] virtio_blk virtio1: [vda] 146232 512-byte logical blocks (74.9 MB/71.4 MiB) [ 2.708820] vda: detected capacity change from 0 to 74870784 [ 2.720535] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.722638] vdb: detected capacity change from 0 to 1073741824 [ 2.727669] libphy: Fixed MDIO Bus: probed [ 2.733089] usbcore: registered new interface driver usbserial_generic [ 2.735035] usbserial: USB Serial support registered for generic [ 2.736654] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.739759] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.741229] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.743398] mousedev: PS/2 mouse device common for all mice [ 2.746539] rtc_cmos 00:05: RTC can wake from S4 [ 2.749297] rtc_cmos 00:05: registered as rtc0 [ 2.750448] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.751171] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.756356] intel_pstate: CPU model not supported [ 2.759517] hid: raw HID events driver (C) Jiri Kosina [ 2.761380] usbcore: registered new interface driver usbhid [ 2.762075] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.763417] usbhid: USB HID core driver [ 2.768046] drop_monitor: Initializing network drop monitor service [ 2.769791] Initializing XFRM netlink socket [ 2.771142] NET: Registered protocol family 10 [ 2.772495] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.773150] Segment Routing with IPv6 [ 2.776676] NET: Registered protocol family 17 [ 2.779123] mpls_gso: MPLS GSO support [ 2.784625] RAS: Correctable Errors collector initialized. [ 2.786874] AVX version of gcm_enc/dec engaged. [ 2.788707] AES CTR mode by8 optimization enabled [ 2.864246] sched_clock: Marking stable (2864221519, 0)->(3666184992, -801963473) [ 2.867596] registered taskstats version 1 [ 2.870447] Loading compiled-in X.509 certificates [ 2.872102] zswap: loaded using pool lzo/zbud [ 2.896780] Key type big_key registered [ 2.907226] Key type encrypted registered [ 2.909041] ima: No TPM chip found, activating TPM-bypass! [ 2.910614] ima: Allocated hash algorithm: sha1 [ 2.911863] ima: No architecture policies found [ 2.913456] evm: Initialising EVM extended attributes: [ 2.914657] evm: security.selinux [ 2.915513] evm: security.ima [ 2.916304] evm: security.capability [ 2.917411] evm: HMAC attrs: 0x1 [ 2.920525] rtc_cmos 00:05: setting system clock to 2026-09-07 14:25:05 UTC (1788791105) [ 2.924646] debug: unmapping init [mem 0xffffffff9d003000-0xffffffff9d1fffff] [ 2.926643] debug: unmapping init [mem 0xffffffff9bd82000-0xffffffff9c058fff] [ 2.934055] Write protecting the kernel read-only data: 28672k [ 2.936357] debug: unmapping init [mem 0xffffffff9a403000-0xffffffff9a5fffff] [ 2.938366] debug: unmapping init [mem 0xffffffff9ad14000-0xffffffff9adfffff] [ 2.965905] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.973123] systemd[1]: Detected virtualization kvm. [ 2.974309] systemd[1]: Detected architecture x86-64. [ 2.975410] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.999641] systemd[1]: No hostname configured. [ 3.000693] systemd[1]: Set hostname to . [ 3.001957] random: systemd: uninitialized urandom read (16 bytes read) [ 3.003600] systemd[1]: Initializing machine ID from random generator. [ 3.026299] random: ln: uninitialized urandom read (6 bytes read) [ 3.114061] random: systemd: uninitialized urandom read (16 bytes read) [ 3.115696] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.119205] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.122199] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.597870] device-mapper: uevent: version 1.0.3 [ 3.599921] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.248756] virtio_net virtio0 ens2: renamed from eth0 [ 4.271688] random: fast init done [ 4.278592] scsi host0: ata_piix [ 4.281112] scsi host1: ata_piix [ 4.282526] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.284613] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.463491] random: crng init done [ 9.476683] random: 7 urandom warning(s) missed due to ratelimiting [ 9.491438] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.256515] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.329162] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.441438] SELinux: Disabled at runtime. [ 15.528388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.542873] systemd[1]: Detected virtualization kvm. [ 15.550591] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.065418] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.081754] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.100144] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.113322] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.120481] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.149413] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.186093] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 18.772874] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.709683] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.994440] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.136536] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.755180] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.880403] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 26.206726] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.087243] NFS: Registering the id_resolver key type [ 27.089421] Key type id_resolver registered [ 27.091353] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg239-client login: [ 42.929178] hrtimer: interrupt took 6188173 ns [ 95.401848] libcfs: loading out-of-tree module taints kernel. [ 95.600114] Key type ._llcrypt registered [ 95.608777] Key type .llcrypt registered [ 96.054440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 96.081210] alg: No test for adler32 (adler32-zlib) [ 97.822671] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 98.992959] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 100.855236] Key type lgssc registered [ 103.057145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 234.297836] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c [ 239.283817] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.631727] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing check_logdir /tmp/testlogs/ [ 255.709491] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing yml_node [ 260.064473] Lustre: lustre-OST0000-osc-ffff9e8e983aa800: disconnect after 23s idle [ 260.298451] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 263.219749] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 265.584239] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 266.960932] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 10:29:28 EDT 2026 [ 284.389165] Lustre: Unmounted lustre-client [ 298.574073] Key type lgssc unregistered [ 298.815898] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.826865] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.859240] LNet: Removed LNI 192.168.202.39@tcp [ 299.989908] Key type .llcrypt unregistered [ 300.004633] Key type ._llcrypt unregistered [ 329.391925] Key type ._llcrypt registered [ 329.393601] Key type .llcrypt registered [ 329.644090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 329.682375] alg: No test for adler32 (adler32-zlib) [ 330.618193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 330.695447] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 330.731992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 334.380610] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 334.525574] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.534744] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.562278] LNet: Removed LNI 192.168.202.39@tcp [ 335.542967] Key type .llcrypt unregistered [ 335.545481] Key type ._llcrypt unregistered [ 342.022427] Key type ._llcrypt registered [ 342.026852] Key type .llcrypt registered [ 343.125637] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:30:44 (1788791444) [ 343.555405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.577621] alg: No test for adler32 (adler32-zlib) [ 344.454692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.537341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.559657] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.584849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.661288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.680941] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 352.345538] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:30:53 (1788791453) [ 352.504285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.537715] LNet: 10303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.585643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.698918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 361.132035] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:31:02 (1788791462) [ 361.305357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.342785] LNet: 10888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.393953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.499760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 368.798859] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:31:09 (1788791469) [ 369.015136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.058533] LNet: 11477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.095817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.195727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 376.492713] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:31:17 (1788791477) [ 376.754369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.774184] LNet: 12063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.795603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.862700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 377.361567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.120524] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:31:25 (1788791485) [ 384.212800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.242302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.318709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.837841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.195444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.804659] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:31:34 (1788791494) [ 393.055520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.096375] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.102735] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 393.179545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.278646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.484954] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:31:42 (1788791502) [ 401.685256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.777482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.855811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 401.922871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.438780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 402.470379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.132577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.193888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.846049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 403.889781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.535939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 404.597535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.465177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 405.524853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.080658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 406.131472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.662308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 406.700692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 407.323311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 407.373346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.080377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 408.132175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.581178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 408.614430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.968806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 409.003403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.362842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 409.435618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 416.149673] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:31:57 (1788791517) [ 416.260309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.336507] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.349592] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 416.391534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.481349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 416.505461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 423.340550] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:32:04 (1788791524) [ 423.579161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.645250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.708840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 423.744588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 431.906243] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:32:13 (1788791533) [ 431.996870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.057450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.147814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 432.202279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 438.262588] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:32:19 (1788791539) [ 438.394195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.451110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.579355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 438.636569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.645755] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:32:26 (1788791546) [ 444.862070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.905417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.995350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 445.035845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 450.904634] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:32:32 (1788791552) [ 451.005673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.027641] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.031410] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 451.061308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.119389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 451.146202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 457.499892] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:32:38 (1788791558) [ 457.684370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.758596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.847795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.128254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.200687] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:32:45 (1788791565) [ 464.464938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.500275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.703171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.893613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.040873] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:32:53 (1788791573) [ 472.188884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.267618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.416379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.581941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.008903] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:33:01 (1788791581) [ 481.177337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.234551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.950455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.446577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.568428] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:33:12 (1788791592) [ 491.721195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.778967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.881629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.941619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.273498] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:33:20 (1788791600) [ 499.705128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.772212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.934101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.999627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 508.202287] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:33:29 (1788791609) [ 508.353017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.399665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.516769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 514.728526] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:33:35 (1788791615) [ 514.831080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.914336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.007238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 521.574536] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:33:42 (1788791622) [ 521.781236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.792945] LNet: 22886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.797781] LNet: 22886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 521.814463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.866661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 522.258512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 528.534925] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:33:49 (1788791629) [ 528.713332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.790217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.911641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 528.986109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 535.484550] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:33:56 (1788791636) [ 535.867040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.931196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.015273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 536.051864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 543.384491] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:34:04 (1788791644) [ 543.660594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.704293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.804511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 543.837870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 550.116591] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:34:11 (1788791651) [ 550.472462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.534385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.600657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.651484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 556.949923] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:34:18 (1788791658) [ 557.069348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.129824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.191270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 557.245984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 557.263741] LNetError: 25834:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 557.298987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 557.330567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 562.888879] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:34:24 (1788791664) [ 563.011356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.092947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.211926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q5K1MO2JtJ [ 563.304934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q5K1MO2JtJ [ 563.370595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q5K1MO2JtJ [ 568.470265] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:34:29 (1788791669) [ 568.815461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.854526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.905214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 568.930649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 568.987232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 574.375596] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:34:35 (1788791675) [ 574.691277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.725982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.837833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 574.886058] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 576.342185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 577.593565] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 579.487529] Key type lgssc registered [ 580.813579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 584.959209] LNetError: 27595:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.39@tcp99 has no available nets [ 587.816480] Key type lgssc unregistered [ 588.172799] LNetError: 28706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.189473] LNet: Removed LNI 192.168.202.39@tcp [ 588.840386] Key type .llcrypt unregistered [ 588.844242] Key type ._llcrypt unregistered [ 595.504137] Key type ._llcrypt registered [ 595.508830] Key type .llcrypt registered [ 597.082853] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:34:57 (1788791697) [ 597.676690] Key type .llcrypt unregistered [ 597.678977] Key type ._llcrypt unregistered [ 604.440861] Key type ._llcrypt registered [ 604.443739] Key type .llcrypt registered [ 604.589745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.606181] alg: No test for adler32 (adler32-zlib) [ 605.488634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.530456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 605.541836] LNet: 29626:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 605.546665] LNet: 29626:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 605.551218] LNet: 29626:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 605.561469] LNet: Message forwarding will be enabled in 2s [ 608.671130] LNet: Message forwarding enabled [ 609.075208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 609.101560] LNet: Message forwarding will stop in 2s to 6s [ 612.771604] LNet: Message forwarding disabled [ 613.560408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 613.589482] LNet: 29724:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 613.595945] LNet: 29724:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 613.606833] LNet: 29724:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 613.634226] LNet: Message forwarding will be enabled in 2s [ 613.798566] LNet: 29737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.410818] Key type .llcrypt unregistered [ 614.412653] Key type ._llcrypt unregistered [ 627.095136] Key type ._llcrypt registered [ 627.113381] Key type .llcrypt registered [ 628.476468] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:35:29 (1788791729) [ 628.887908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.901448] alg: No test for adler32 (adler32-zlib) [ 629.906667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.022655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 636.270568] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:35:37 (1788791737) [ 636.575394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.604169] LNet: 31145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.642206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.707206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 636.738122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 636.771430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 636.798270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 636.821340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 636.848989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 636.874501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 636.896637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 636.926886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 636.949452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 636.970383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 636.996702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 637.018981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 637.041481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 637.068251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 637.098383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 637.126722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 637.159888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 637.188541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 637.208745] LNetError: 31188:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 643.263915] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:35:44 (1788791744) [ 643.356339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.414580] LNet: 31750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.482790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.632931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 651.309129] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:35:52 (1788791752) [ 653.493547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg239-server [ 655.245630] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:35:56 (1788791756) [ 655.644847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.694857] LNet: 32690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.777916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.873331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.890794] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 656.842314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[40-44]@tcp [ 656.882801] LNet: 32824:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 656.937279] LNetError: 32694:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 656.959152] LNetError: 32694:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.42@tcp has gone from up to down [ 664.142756] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:36:05 (1788791765) [ 666.188840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg239-server [ 668.026528] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:36:09 (1788791769) [ 668.144805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.194974] LNet: 33753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.209577] LNetError: 33753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.228853] LNet: Removed LNI 192.168.202.39@tcp [ 668.266165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.348299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.382670] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 669.294709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[40-45/2]@tcp [ 669.311629] LNet: 33877:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 669.318333] LNet: 33877:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 669.339676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[40-45/2]@tcp [ 669.347437] LNetError: 33763:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 669.374535] LNetError: 33763:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 669.393361] LNetError: 33763:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.44@tcp has gone from up to down [ 669.404508] LNetError: 33763:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 676.274323] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:36:17 (1788791777) [ 676.505265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.556790] LNet: 34468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.566621] LNetError: 34468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.584917] LNet: Removed LNI 192.168.202.39@tcp [ 676.641320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.736243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 676.827907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.873549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.969547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.042187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.099576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.169665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 677.275912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.330894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.413085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.504505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.536258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.598603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 677.692619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.730594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.801364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.880454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.943720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.023989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 678.206915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.260870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.328294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.502312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.567906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.663505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 678.868441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.951566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.048309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.177286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.213849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.264845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 679.341573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.407433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.484843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.355183] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:36:27 (1788791787) [ 688.536218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg239-server [ 690.250811] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:36:31 (1788791791) [ 691.676737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg239-server [ 693.722816] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:36:34 (1788791794) [ 693.811411] LNet: 35864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.816615] LNet: 35864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 694.484200] Key type .llcrypt unregistered [ 694.487171] Key type ._llcrypt unregistered [ 700.213581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.417046] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.442189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.450193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.104435] Key type ._llcrypt registered [ 703.106700] Key type .llcrypt registered [ 703.417861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.444946] alg: No test for adler32 (adler32-zlib) [ 704.418330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.545503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.589932] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 705.599406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.641620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.141853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 708.166490] LNet: Removed LNI 10.1.2.3@tcp [ 708.381384] LNet: 36597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.395455] LNetError: 36597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.413315] LNet: Removed LNI 192.168.202.39@tcp [ 709.159451] Key type .llcrypt unregistered [ 709.161769] Key type ._llcrypt unregistered [ 722.104971] Key type ._llcrypt registered [ 722.108419] Key type .llcrypt registered [ 723.649987] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:37:04 (1788791824) [ 724.253416] Key type .llcrypt unregistered [ 724.255826] Key type ._llcrypt unregistered [ 730.491980] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.659523] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.673190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.681954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.331285] Key type ._llcrypt registered [ 733.334183] Key type .llcrypt registered [ 733.591037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.632727] alg: No test for adler32 (adler32-zlib) [ 734.508967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.552118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.584577] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 735.754071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 735.793403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.007322] LNet: 38094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.017402] LNetError: 38094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.032953] LNet: Removed LNI 10.1.2.3@tcp [ 739.058713] LNet: Removed LNI 192.168.202.39@tcp [ 740.384790] Key type .llcrypt unregistered [ 740.395783] Key type ._llcrypt unregistered [ 760.221086] Key type ._llcrypt registered [ 760.222893] Key type .llcrypt registered [ 762.205430] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:37:42 (1788791862) [ 762.679335] Key type .llcrypt unregistered [ 762.686343] Key type ._llcrypt unregistered [ 774.126123] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.309325] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.312028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.315866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.090095] Key type ._llcrypt registered [ 777.096247] Key type .llcrypt registered [ 777.290982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.303778] alg: No test for adler32 (adler32-zlib) [ 778.199679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.276253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 778.306943] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 781.514412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.549772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.064833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 785.097167] LNet: Removed LNI 10.1.2.3@tcp [ 785.133434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 785.177973] LNet: Removed LNI 10.1.2.31@tcp [ 785.191384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.250578] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.135797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 786.164252] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 787.708391] LNet: 40025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.722648] LNetError: 40025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 787.728052] LNet: Removed LNI 10.1.2.31@tcp [ 787.752646] LNet: Removed LNI 10.1.2.3@tcp [ 788.491467] Key type .llcrypt unregistered [ 788.496293] Key type ._llcrypt unregistered [ 802.900131] Key type ._llcrypt registered [ 802.905144] Key type .llcrypt registered [ 804.370449] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:38:25 (1788791905) [ 804.766847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.778794] alg: No test for adler32 (adler32-zlib) [ 805.662751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.714980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.737405] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 806.409493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.424842] LNet: 40983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.430399] LNetError: 40983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.444682] LNet: Removed LNI 192.168.202.39@tcp [ 806.458873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.507785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.39@tcp [ 806.532480] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 807.516402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.39@tcp [ 807.544612] LNet: Removed LNI 192.168.202.39@tcp [ 807.598837] LNet: 41118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.332827] Key type .llcrypt unregistered [ 808.340083] Key type ._llcrypt unregistered [ 821.585260] Key type ._llcrypt registered [ 821.603515] Key type .llcrypt registered [ 823.220354] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:38:44 (1788791924) [ 825.692559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg239-server [ 827.646200] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:38:48 (1788791928) [ 828.315466] Key type .llcrypt unregistered [ 828.318960] Key type ._llcrypt unregistered [ 835.264398] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.485119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.498174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.514862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.230901] Key type ._llcrypt registered [ 837.233917] Key type .llcrypt registered [ 837.449508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.457401] alg: No test for adler32 (adler32-zlib) [ 838.419498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.517674] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 839.654932] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 842.101978] LNet: 42953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.105935] LNetError: 42953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.108896] LNet: Removed LNI 192.168.202.39@tcp1 [ 842.115497] LNet: Removed LNI 10.1.2.3@tcp2 [ 843.010474] Key type .llcrypt unregistered [ 843.011990] Key type ._llcrypt unregistered [ 857.513350] Key type ._llcrypt registered [ 857.515796] Key type .llcrypt registered [ 859.018056] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:39:19 (1788791959) [ 859.590704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.608964] alg: No test for adler32 (adler32-zlib) [ 860.548814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.644528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.667157] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 861.660618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.689475] LNet: 43928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.700712] LNetError: 43928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.726878] LNet: Removed LNI 192.168.202.39@tcp [ 861.756689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.815799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 861.862679] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 869.680357] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:39:30 (1788791970) [ 869.998709] LNet: 44519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.013716] LNetError: 44519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.033282] LNet: Removed LNI 192.168.202.39@tcp [ 870.673437] Key type .llcrypt unregistered [ 870.679853] Key type ._llcrypt unregistered [ 877.162657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.248275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.262356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.265175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.046619] Key type ._llcrypt registered [ 880.054395] Key type .llcrypt registered [ 880.314558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.324691] alg: No test for adler32 (adler32-zlib) [ 881.207475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.265565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.284290] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 882.034816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 882.057924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.075531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.118979] LNet: 45266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.131380] LNetError: 45266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.142981] LNet: Removed LNI 10.1.2.3@tcp [ 884.165738] LNet: Removed LNI 192.168.202.39@tcp [ 884.208498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.393596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.449036] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 884.456197] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.145256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.179575] LNet: 45293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.188540] LNetError: 45293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.194248] LNetError: 45293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.199576] LNet: Removed LNI 10.1.2.3@tcp [ 885.221576] LNet: Removed LNI 192.168.202.39@tcp [ 885.259889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.380763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.427606] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 885.444865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.363470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 886.385576] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 887.453373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.475962] LNet: 45456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.488597] LNetError: 45456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 887.499820] LNetError: 45456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.511198] LNet: Removed LNI 10.1.2.3@tcp [ 887.512750] LNet: Removed LNI 192.168.202.39@tcp [ 887.547294] LNet: Removed LNI 192.168.202.39@tcp2 [ 887.566445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.654314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 887.674579] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 887.681986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 887.689542] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 897.201474] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:39:57 (1788791997) [ 897.572995] LNet: 46055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.594199] LNetError: 46055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.607023] LNetError: 46055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 897.635939] LNet: Removed LNI 10.1.2.3@tcp [ 897.640503] LNet: Removed LNI 192.168.202.39@tcp [ 897.666235] LNet: Removed LNI 192.168.202.39@tcp2 [ 898.620865] Key type .llcrypt unregistered [ 898.627174] Key type ._llcrypt unregistered [ 906.632419] Key type ._llcrypt registered [ 906.634117] Key type .llcrypt registered [ 906.828577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.870792] alg: No test for adler32 (adler32-zlib) [ 907.860655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.930625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.957546] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 909.092837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.127472] LNet: 46456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.139373] LNetError: 46456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.164399] LNet: Removed LNI 192.168.202.39@tcp [ 909.184091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.258571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.288694] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 909.937433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.951846] LNet: 46486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.954822] LNetError: 46486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.970443] LNet: Removed LNI 192.168.202.39@tcp [ 909.982732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.038876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 910.055442] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 910.292434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.310515] LNet: Removed LNI 192.168.202.39@tcp [ 910.355292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.436991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 910.461993] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 910.844261] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.899137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.910653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.922602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.374765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.398377] LNet: 46594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.405958] LNet: 46594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 913.411628] LNetError: 46594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.416343] LNetError: 46594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 913.428647] LNet: Removed LNI 192.168.202.39@tcp [ 913.449471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.496663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.514958] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 921.580201] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:40:22 (1788792022) [ 922.117076] LNet: 47192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.124638] LNetError: 47192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.140742] LNet: Removed LNI 192.168.202.39@tcp [ 922.825988] Key type .llcrypt unregistered [ 922.835019] Key type ._llcrypt unregistered [ 930.019042] Key type ._llcrypt registered [ 930.023897] Key type .llcrypt registered [ 930.249274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.274237] alg: No test for adler32 (adler32-zlib) [ 931.166455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.200935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.223215] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 931.876337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.890510] LNet: 47594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.895702] LNetError: 47594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.919152] LNet: Removed LNI 192.168.202.39@tcp [ 931.938949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.009329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 932.033668] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 941.493313] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:40:42 (1788792042) [ 941.611174] LNet: 48182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.617574] LNetError: 48182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.631924] LNet: Removed LNI 192.168.202.39@tcp [ 942.318561] Key type .llcrypt unregistered [ 942.321685] Key type ._llcrypt unregistered [ 952.486606] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 952.793611] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 952.807376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 952.819933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.698126] Key type ._llcrypt registered [ 955.704727] Key type .llcrypt registered [ 956.014111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.042787] alg: No test for adler32 (adler32-zlib) [ 957.101132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.178605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.195970] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 958.611356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.672090] LNet: 48643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.687148] LNetError: 48643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.717738] LNet: Removed LNI 192.168.202.39@tcp [ 958.770686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.908672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 958.962326] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 969.739915] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:41:09 (1788792069) [ 969.858832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.875280] LNet: 49238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.890518] LNetError: 49238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.911258] LNet: Removed LNI 192.168.202.39@tcp [ 969.980913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.133717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 975.815188] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:41:17 (1788792077) [ 975.979371] LNet: 49820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.699614] Key type .llcrypt unregistered [ 976.702276] Key type ._llcrypt unregistered [ 983.733632] Key type ._llcrypt registered [ 983.736315] Key type .llcrypt registered [ 984.024583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.051354] alg: No test for adler32 (adler32-zlib) [ 984.954569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.048337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.081117] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 985.919121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.938601] LNet: 50226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.943795] LNetError: 50226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.957632] LNet: Removed LNI 192.168.202.39@tcp [ 985.975601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.035859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 986.066213] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 992.520345] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:41:33 (1788792093) [ 992.741654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.765291] LNet: 50820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.771220] LNetError: 50820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.780485] LNet: Removed LNI 192.168.202.39@tcp [ 992.804799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.871638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 999.644142] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:41:40 (1788792100) [ 1001.415289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1003.572543] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:41:44 (1788792104) [ 1005.855885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1007.981399] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:41:48 (1788792108) [ 1009.908960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1012.027180] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:41:52 (1788792112) [ 1013.804983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1016.285676] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:41:57 (1788792117) [ 1018.468351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1020.315626] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:42:01 (1788792121) [ 1020.638163] LNet: 52874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.254491] Key type .llcrypt unregistered [ 1021.256667] Key type ._llcrypt unregistered [ 1029.327430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.592645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1029.600567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1029.616409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.340658] Key type ._llcrypt registered [ 1032.342254] Key type .llcrypt registered [ 1032.625308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.672632] alg: No test for adler32 (adler32-zlib) [ 1033.698695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.864395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1033.912701] LNet: Added LNI 192.168.202.39@tcp [8/512/0/180] [ 1033.929105] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1033.934124] LNet: 53197:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1034.125695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.154600] LNet: 53216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.167805] LNetError: 53216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1034.174626] LNet: Removed LNI 10.1.2.3@tcp [ 1034.179833] LNetError: 53216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.208685] LNet: Removed LNI 192.168.202.39@tcp [ 1034.247812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.333230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1034.379737] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1034.416878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.428535] LNet: 53240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.431685] LNetError: 53240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.440704] LNet: Removed LNI 192.168.202.39@tcp [ 1034.449633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.502725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1034.530148] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1034.537189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.542101] LNet: 53248:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1034.661719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.679947] LNet: 53266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.683780] LNetError: 53266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1034.686688] LNet: Removed LNI 10.1.2.3@tcp [ 1034.691021] LNetError: 53266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.705714] LNet: Removed LNI 192.168.202.39@tcp [ 1034.716858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.770963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1034.791678] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1034.800852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.804799] LNet: 53274:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1034.995636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.021866] LNet: 53300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.037365] LNetError: 53300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.045370] LNet: Removed LNI 10.1.2.3@tcp [ 1035.049452] LNetError: 53300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.075846] LNet: Removed LNI 192.168.202.39@tcp [ 1035.101549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.184701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1035.231391] LNet: Added LNI 192.168.202.39@tcp [8/256/0/182] [ 1035.238360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1035.335460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.352508] LNet: 53328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.356628] LNetError: 53328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.361278] LNet: Removed LNI 10.1.2.3@tcp [ 1035.363959] LNetError: 53328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.375484] LNet: Removed LNI 192.168.202.39@tcp [ 1035.393314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.465425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1035.496366] LNet: Added LNI 192.168.202.39@tcp [8/256/0/182] [ 1035.505991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1035.507974] LNet: 53336:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1035.571810] LNet: Removed LNI 10.1.2.3@tcp [ 1035.579693] LNet: Removed LNI 192.168.202.39@tcp [ 1036.221350] Key type .llcrypt unregistered [ 1036.226862] Key type ._llcrypt unregistered [ 1048.787813] Key type ._llcrypt registered [ 1048.791183] Key type .llcrypt registered [ 1050.040068] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:42:31 (1788792151) [ 1051.570307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1053.301527] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:42:34 (1788792154) [ 1053.895313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.924334] alg: No test for adler32 (adler32-zlib) [ 1054.897631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.943672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1054.969425] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1055.882726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.39 [ 1055.903117] LNet: 54604:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1055.912855] LNet: Removed LNI 192.168.202.39@tcp [ 1062.123641] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:42:43 (1788792163) [ 1062.218700] LNet: 55165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.232454] LNetError: 55165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.284537] LNet: Removed LNI 192.168.202.39@tcp [ 1063.068309] Key type .llcrypt unregistered [ 1063.069787] Key type ._llcrypt unregistered [ 1069.756143] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1070.073583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1072.978127] Key type ._llcrypt registered [ 1072.979853] Key type .llcrypt registered [ 1073.201786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.226669] alg: No test for adler32 (adler32-zlib) [ 1074.615059] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1081.120437] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:43:02 (1788792182) [ 1081.334875] LNet: 56092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.340886] LNetError: 56092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1081.356892] LNet: Removed LNI 10.1.2.3@tcp [ 1081.909995] Key type .llcrypt unregistered [ 1081.912055] Key type ._llcrypt unregistered [ 1087.548148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1087.754667] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1087.770597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1087.775366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1090.489134] Key type ._llcrypt registered [ 1090.495432] Key type .llcrypt registered [ 1090.680672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.696877] alg: No test for adler32 (adler32-zlib) [ 1091.665135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.497418] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:43:19 (1788792199) [ 1100.260158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1101.813708] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:43:23 (1788792203) [ 1101.955660] LNet: 57275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.963980] LNetError: 57275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.986691] LNet: Removed LNI 10.1.2.3@tcp [ 1102.750709] Key type .llcrypt unregistered [ 1102.754294] Key type ._llcrypt unregistered [ 1109.760167] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1110.084654] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1110.087936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1110.099535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1113.005746] Key type ._llcrypt registered [ 1113.008721] Key type .llcrypt registered [ 1113.256343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.275867] alg: No test for adler32 (adler32-zlib) [ 1114.322790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.445705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1122.603388] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:43:43 (1788792223) [ 1123.174221] LNet: 58513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.191232] LNetError: 58513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1123.224923] LNet: Removed LNI 10.1.2.3@tcp [ 1123.941667] Key type .llcrypt unregistered [ 1123.945797] Key type ._llcrypt unregistered [ 1131.321632] Key type ._llcrypt registered [ 1131.328920] Key type .llcrypt registered [ 1131.760451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.777899] alg: No test for adler32 (adler32-zlib) [ 1132.730739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1132.778567] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1141.741197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1142.794277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_interrupt -r 1 [ 1142.887654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_interrupt -r 1 [ 1142.930893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1142.958908] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1143.007317] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1143.330366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.363587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.390856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.423205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.488448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.527436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.575769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.656666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.690127] LNet: 58955:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.725392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.790737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1143.837306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1143.913813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.974115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.041657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.119956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.895686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_dropped -r 1 [ 1144.930108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_dropped -r 1 [ 1144.972828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1144.987593] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1144.992575] LNet: Skipped 1 previous similar message [ 1145.223845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.259818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.297343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.346280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.398358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.473920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.541883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.590678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.603511] LNet: 59064:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.643741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.691497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.729779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.771547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.810876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.853370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.902687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.635734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_aborted -r 1 [ 1146.683551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_aborted -r 1 [ 1146.730753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1146.764690] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1146.771827] LNet: Skipped 1 previous similar message [ 1147.054833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.118290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.194084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.267055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.347201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.408886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.472772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.516748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.560800] LNet: 59173:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.582556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.606976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.631295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.650760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.679743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.735054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.773739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1148.602382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_no_route -r 1 [ 1148.723342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_no_route -r 1 [ 1148.839360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1148.877537] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1149.362746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.430438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.470435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.516242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.565993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.635276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.701205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.754792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.802305] LNet: 59283:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.841856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.928448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.050089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.113277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.199642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.252701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.299533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.948949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_timeout -r 1 [ 1150.988582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_timeout -r 1 [ 1151.054233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1151.332808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.384865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.423464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.465653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.512117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.539822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.598555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.655837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.708115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.774802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.846638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.898915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.983939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.041769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.094551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1152.755831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_error -r 1 [ 1152.829422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_error -r 1 [ 1152.863034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1152.899125] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1152.906320] LNet: Skipped 1 previous similar message [ 1153.148701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.177805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.206771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.234299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.262671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.302212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.343297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.406902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.467657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.520373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.547732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.577681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.603492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.630209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.659419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1154.036972] LNet: 59546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.042692] LNetError: 59546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.054471] LNet: Removed LNI 192.168.202.39@tcp [ 1154.580636] Key type .llcrypt unregistered [ 1154.586705] Key type ._llcrypt unregistered [ 1173.373990] Key type ._llcrypt registered [ 1173.376514] Key type .llcrypt registered [ 1174.805189] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:44:35 (1788792275) [ 1176.000462] Key type .llcrypt unregistered [ 1176.002386] Key type ._llcrypt unregistered [ 1183.364867] Key type ._llcrypt registered [ 1183.368441] Key type .llcrypt registered [ 1183.542334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1183.564247] alg: No test for adler32 (adler32-zlib) [ 1184.627948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1184.660763] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1193.594540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1204.315518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.389567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1204.427684] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1205.857880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_interrupt -r 1 [ 1205.883500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_interrupt -r 1 [ 1205.911135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_interrupt -r 1 [ 1205.961742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_interrupt -r 1 [ 1206.037501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_interrupt -r 1 [ 1206.111693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_interrupt -r 1 [ 1206.171498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_interrupt -r 1 [ 1206.234915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_interrupt -r 1 [ 1206.296811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1206.351458] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1206.362173] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1206.654424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.692682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.745184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.792191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.844642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.900370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.978150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.016569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.036514] LNet: 61019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.052664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.099913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.144824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.212252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.293434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.358905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.392605] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1207.407750] LNet: Skipped 3 previous similar messages [ 1207.432760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1207.945832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_dropped -r 1 [ 1207.975951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_dropped -r 1 [ 1208.003855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_dropped -r 1 [ 1208.028425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_dropped -r 1 [ 1208.053959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_dropped -r 1 [ 1208.096149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_dropped -r 1 [ 1208.151713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_dropped -r 1 [ 1208.182877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_dropped -r 1 [ 1208.212193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1208.412494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.433569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.464565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.499496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.569858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1208.618827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1208.672593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1208.725797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1208.768959] LNet: 61141:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.792717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.828831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.916655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.986589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.077662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.176578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.211536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.072709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_aborted -r 1 [ 1210.118548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_aborted -r 1 [ 1210.161876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_aborted -r 1 [ 1210.197287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_aborted -r 1 [ 1210.238973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_aborted -r 1 [ 1210.298214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_aborted -r 1 [ 1210.354827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_aborted -r 1 [ 1210.441770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_aborted -r 1 [ 1210.474751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1210.498937] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1210.510660] LNet: Skipped 3 previous similar messages [ 1210.835210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.871687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.914138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.953989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.995035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.051188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.129430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.175632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.246958] LNet: 61262:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.283827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.315487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.381857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.458177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.514826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.580573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.625408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1212.531808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_no_route -r 1 [ 1212.618634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_no_route -r 1 [ 1212.687954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_no_route -r 1 [ 1212.762841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_no_route -r 1 [ 1212.875717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_no_route -r 1 [ 1212.940610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_no_route -r 1 [ 1212.986439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_no_route -r 1 [ 1213.037815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_no_route -r 1 [ 1213.088565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1213.116398] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1213.119502] LNet: Skipped 2 previous similar messages [ 1213.482591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.531537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1213.562906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.580966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.598602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.633457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.681435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.729969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.784996] LNet: 61384:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.832906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.906453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.963904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.033418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.081353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.157677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.230591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.836563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_timeout -r 1 [ 1214.872257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_timeout -r 1 [ 1214.907124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_timeout -r 1 [ 1214.940644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_timeout -r 1 [ 1214.975709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_timeout -r 1 [ 1215.012829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_timeout -r 1 [ 1215.066701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_timeout -r 1 [ 1215.118601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_timeout -r 1 [ 1215.166485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1215.540605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.624300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.686702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.769678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.836151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.894650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.944200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.976441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.018570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.048668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.078582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.121443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.171421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.215831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.259953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.879957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_error -r 1 [ 1216.906915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_error -r 1 [ 1216.928978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_error -r 1 [ 1216.962511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_error -r 1 [ 1217.022967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_error -r 1 [ 1217.125401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_error -r 1 [ 1217.196959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_error -r 1 [ 1217.261797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_error -r 1 [ 1217.307577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1217.325481] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1217.336433] LNet: Skipped 6 previous similar messages [ 1217.639345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.676295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.722759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.791136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.852108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.906327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.971925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.036336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.074744] LNet: 61627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.084805] LNet: 61627:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1218.120532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.157318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.196639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.233994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.294378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.329441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.372215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.735806] LNet: 61707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.740868] LNetError: 61707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.799303] LNet: Removed LNI 192.168.202.39@tcp [ 1224.805358] LNetError: 61707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.823937] LNet: Removed LNI 192.168.202.39@tcp1 [ 1225.338905] Key type .llcrypt unregistered [ 1225.343591] Key type ._llcrypt unregistered [ 1248.903874] Key type ._llcrypt registered [ 1248.908898] Key type .llcrypt registered [ 1250.501330] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:45:51 (1788792351) [ 1251.399657] Key type .llcrypt unregistered [ 1251.401346] Key type ._llcrypt unregistered [ 1257.936602] Key type ._llcrypt registered [ 1257.950554] Key type .llcrypt registered [ 1258.218987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.245602] alg: No test for adler32 (adler32-zlib) [ 1259.230915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.283030] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1267.476604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1268.367529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_dropped -r 1 [ 1268.412412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_dropped -r 1 [ 1268.448550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1268.476238] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1268.759439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.805150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.846224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.905895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.962360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.017884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.063231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.097153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.125528] LNet: 62961:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.143991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.168269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.211331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.263676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.307228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.353611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.389957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.050285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_error -r 1 [ 1270.092635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_error -r 1 [ 1270.123320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1270.148162] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1270.317601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.349483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.374482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.399732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.441738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.481722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.509639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.538968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.554569] LNet: 63071:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.574629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.604671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.632553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.665369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.696976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.725973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.755050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1271.139491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_timeout -r 1 [ 1271.160976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_timeout -r 1 [ 1271.185940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1271.202685] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1271.458279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.504851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.553359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.627869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.701310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.760436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.818721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.854714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.876097] LNet: 63180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.889926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.931625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.976869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.005113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.026625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.078448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.125464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1272.524340] LNet: 63225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.536323] LNetError: 63225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.581531] LNet: Removed LNI 192.168.202.39@tcp [ 1273.126675] Key type .llcrypt unregistered [ 1273.129722] Key type ._llcrypt unregistered [ 1291.805987] Key type ._llcrypt registered [ 1291.812685] Key type .llcrypt registered [ 1293.473963] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:46:34 (1788792394) [ 1294.854555] Key type .llcrypt unregistered [ 1294.860805] Key type ._llcrypt unregistered [ 1302.036306] Key type ._llcrypt registered [ 1302.046069] Key type .llcrypt registered [ 1302.405344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.450848] alg: No test for adler32 (adler32-zlib) [ 1303.359354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1303.412165] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1312.699375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1325.254496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.339116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.412697] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1327.651611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_dropped -r 1 [ 1327.691769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e remote_dropped -r 1 [ 1327.742953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_dropped -r 1 [ 1327.797968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e remote_dropped -r 1 [ 1327.841612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e remote_dropped -r 1 [ 1327.893522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e remote_dropped -r 1 [ 1327.945760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e remote_dropped -r 1 [ 1328.001923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e remote_dropped -r 1 [ 1328.071646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1328.098111] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1328.117033] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.139@tcp [ 1328.529510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.593678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.654409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.726678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.790508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.851416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.885161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.934263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.956324] LNet: 64699:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.986970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.043659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.104060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.184325] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1329.191312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.209778] LNet: Skipped 3 previous similar messages [ 1329.284664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.378032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.475119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.557597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.404245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_error -r 1 [ 1330.426201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e remote_error -r 1 [ 1330.444166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_error -r 1 [ 1330.463692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e remote_error -r 1 [ 1330.492358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e remote_error -r 1 [ 1330.513151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e remote_error -r 1 [ 1330.542905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e remote_error -r 1 [ 1330.623045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e remote_error -r 1 [ 1330.688518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1330.727836] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1331.161608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.205373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.263979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.312565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.363412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.449945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.528845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.625791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.683133] LNet: 64823:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.725052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.813283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.931303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.974991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.053633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.119628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.199188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1332.252567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.887770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_timeout -r 1 [ 1332.970034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e remote_timeout -r 1 [ 1333.028395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_timeout -r 1 [ 1333.097729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e remote_timeout -r 1 [ 1333.171826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e remote_timeout -r 1 [ 1333.242594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e remote_timeout -r 1 [ 1333.314713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e remote_timeout -r 1 [ 1333.379231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e remote_timeout -r 1 [ 1333.438963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.139@tcp [ 1333.458521] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1333.890581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.943314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.003700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.067517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.124857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.174348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.260456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.304329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.334466] LNet: 64946:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.368618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.470783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.538035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.601924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.663341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.710783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.762353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.814834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1340.903629] LNet: 65028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.910782] LNetError: 65028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.919263] LNet: Removed LNI 192.168.202.39@tcp [ 1341.927414] LNetError: 65028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.949945] LNet: Removed LNI 192.168.202.39@tcp1 [ 1342.458609] Key type .llcrypt unregistered [ 1342.460736] Key type ._llcrypt unregistered [ 1358.416139] Key type ._llcrypt registered [ 1358.418069] Key type .llcrypt registered [ 1359.622359] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:47:40 (1788792460) [ 1362.548746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1364.878551] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:47:45 (1788792465) [ 1366.329795] Key type .llcrypt unregistered [ 1366.334166] Key type ._llcrypt unregistered [ 1376.621557] Key type ._llcrypt registered [ 1376.624263] Key type .llcrypt registered [ 1376.791200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1376.808654] alg: No test for adler32 (adler32-zlib) [ 1378.108694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1378.211228] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1389.305538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1390.200985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e network_timeout -r 1 [ 1390.233791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e network_timeout -r 1 [ 1390.310441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1390.369676] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1390.626192] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1390.693620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1390.736992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1390.808845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1390.876766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1390.938219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1390.998554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1391.039079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1391.094824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1391.137156] LNet: 66580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.197388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1391.234429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1391.300523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.357449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.438899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1391.481429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1391.779837] LNet: 66623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.787312] LNetError: 66623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.832669] LNet: Removed LNI 192.168.202.39@tcp [ 1393.359078] Key type .llcrypt unregistered [ 1393.360696] Key type ._llcrypt unregistered [ 1415.858547] Key type ._llcrypt registered [ 1415.860705] Key type .llcrypt registered [ 1416.199748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1416.231275] alg: No test for adler32 (adler32-zlib) [ 1417.200788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1417.224781] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1428.849220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1442.157282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.204514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.234228] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1444.635734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e network_timeout -r 1 [ 1444.685067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e network_timeout -r 1 [ 1444.718608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e network_timeout -r 1 [ 1444.765298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e network_timeout -r 1 [ 1444.830569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e network_timeout -r 1 [ 1444.863898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e network_timeout -r 1 [ 1444.899366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e network_timeout -r 1 [ 1444.950429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e network_timeout -r 1 [ 1444.999990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1445.017596] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1445.284909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1445.328878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1445.379923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1445.418472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1445.461381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1445.517909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1445.568569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1445.611296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1445.661959] LNet: 67510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1445.692335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1445.759644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1445.857349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.920088] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1445.922524] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.139@tcp [ 1445.934196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.984340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1446.023284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1450.994264] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1451.547724] LNet: 67588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.564305] LNetError: 67588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.582341] LNet: Removed LNI 192.168.202.39@tcp [ 1451.608703] LNet: Removed LNI 192.168.202.39@tcp1 [ 1452.305750] Key type .llcrypt unregistered [ 1452.312735] Key type ._llcrypt unregistered [ 1468.312261] Key type ._llcrypt registered [ 1468.314244] Key type .llcrypt registered [ 1469.579332] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:49:30 (1788792570) [ 1469.949145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.970436] alg: No test for adler32 (adler32-zlib) [ 1470.887341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.936531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.964542] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1471.723915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.746549] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1472.702424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 1472.760184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1472.793691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1472.826513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1472.854671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1472.889651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1472.913748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.39@tcp [ 1472.928864] LNetError: 68688:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.39@tcp to recovery [ 1472.994325] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1472.994503] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1474.015364] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1474.235965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1478.111515] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1478.982115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1479.019901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1479.072381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.087230] LNet: 68785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.092871] LNetError: 68785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.160016] LNet: Removed LNI 192.168.202.39@tcp [ 1480.162157] LNetError: 68785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.199849] LNet: Removed LNI 192.168.202.39@tcp1 [ 1480.237251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.305714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.331595] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1480.933725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1480.957425] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1481.684905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 1481.715425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.770502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1481.821363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1481.883921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1481.928415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1481.969676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.023400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.39@tcp [ 1482.035819] LNetError: 69058:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.39@tcp to recovery [ 1482.335778] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1485.594649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1488.480661] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1488.502753] LNet: Skipped 1 previous similar message [ 1493.772698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.829833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1493.890326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1501.500656] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:50:02 (1788792602) [ 1501.701277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.750362] LNet: 69774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.763233] LNetError: 69774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.782210] LNet: Removed LNI 192.168.202.39@tcp [ 1501.806948] LNet: Removed LNI 192.168.202.39@tcp1 [ 1501.882393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.005359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.066932] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1502.904717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.924391] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1503.754934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 1503.817466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1503.846724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1503.876949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1503.906419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1503.935856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.962234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.39@tcp --health 0 [ 1504.051642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.991335] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.39@tcp [ 1504.991776] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1505.023887] LNet: Skipped 1 previous similar message [ 1519.118137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.439225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.488105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.529180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.39@tcp --health 500 [ 1519.633867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.581613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.723080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.774709] LNet: 70260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.785386] LNetError: 70260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.796705] LNetError: 70260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1526.811964] LNet: Removed LNI 192.168.202.39@tcp [ 1526.823260] LNet: Removed LNI 192.168.202.39@tcp1 [ 1526.850223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.983775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.022087] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1528.306700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.333712] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1529.546131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 1529.587607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1529.672977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1529.711675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1529.745935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1529.787139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1529.830295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1529.870285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.39@tcp --health 0 [ 1530.975770] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1530.984360] LNet: Skipped 3 previous similar messages [ 1531.318622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.852925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.888091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.979623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1550.036305] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:50:50 (1788792650) [ 1550.546099] LNet: 71309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.560826] LNetError: 71309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.575893] LNetError: 71309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1550.586091] LNet: Removed LNI 192.168.202.39@tcp [ 1550.599467] LNet: Removed LNI 192.168.202.39@tcp1 [ 1551.274877] Key type .llcrypt unregistered [ 1551.278966] Key type ._llcrypt unregistered [ 1559.068589] Key type ._llcrypt registered [ 1559.070988] Key type .llcrypt registered [ 1559.243179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1559.280582] alg: No test for adler32 (adler32-zlib) [ 1560.293377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1560.338458] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1572.821306] Lustre: DEBUG MARKER: Initial discovery [ 1572.927708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.139@tcp [ 1575.314923] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1575.606774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1575.658058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.139@tcp [ 1575.711215] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1575.738226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1575.783106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.139@tcp -r 1 -m GET -l 3 [ 1575.856730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.139@tcp -r 1 -m GET -e local_error [ 1575.891620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.139@tcp -r 1 -m PUT -l 6 [ 1576.736873] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1577.297214] Lustre: DEBUG MARKER: Force oleg239-client.virtnet to discover 192.168.202.139@tcp (in background) [ 1577.643939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.139@tcp [ 1579.357229] Lustre: DEBUG MARKER: Force oleg239-server to discover 192.168.202.39@tcp [ 1580.769251] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1581.728380] Lustre: DEBUG MARKER: Wait for 72079 [ 1583.145840] Lustre: DEBUG MARKER: Finished wait on 72079 [ 1583.206660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1583.252105] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1583.252764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1583.553154] LNet: 72531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.557266] LNetError: 72531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.623239] LNet: Removed LNI 192.168.202.39@tcp [ 1585.006574] Key type .llcrypt unregistered [ 1585.009287] Key type ._llcrypt unregistered [ 1601.718618] Key type ._llcrypt registered [ 1601.721887] Key type .llcrypt registered [ 1603.023522] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:51:44 (1788792704) [ 1603.775307] Key type .llcrypt unregistered [ 1603.779156] Key type ._llcrypt unregistered [ 1612.509184] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1612.660311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1612.671617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1612.679286] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1615.475810] Key type ._llcrypt registered [ 1615.481395] Key type .llcrypt registered [ 1615.734415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.764200] alg: No test for adler32 (adler32-zlib) [ 1616.670556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.744700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.782769] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1618.194403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1618.219388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1631.206705] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:52:12 (1788792732) [ 1631.338460] LNet: 74607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.348904] LNetError: 74607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1631.361972] LNet: Removed LNI 10.1.2.3@tcp [ 1631.383186] LNet: Removed LNI 192.168.202.39@tcp [ 1632.162018] Key type .llcrypt unregistered [ 1632.167699] Key type ._llcrypt unregistered [ 1641.423075] Key type ._llcrypt registered [ 1641.425123] Key type .llcrypt registered [ 1641.686728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.727061] alg: No test for adler32 (adler32-zlib) [ 1642.671633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.746480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.762875] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1643.459630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1643.489651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1644.779894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.39@tcp [ 1645.855672] LNetError: Unexpected error -22 connecting to 192.168.202.39@tcp at host 192.168.202.39:988 [ 1651.320900] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:52:32 (1788792752) [ 1651.440648] LNet: 75744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.464845] LNetError: 75744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1651.478327] LNet: Removed LNI 10.1.2.3@tcp [ 1651.516724] LNet: Removed LNI 192.168.202.39@tcp [ 1652.346641] Key type .llcrypt unregistered [ 1652.350347] Key type ._llcrypt unregistered [ 1659.551518] Key type ._llcrypt registered [ 1659.554946] Key type .llcrypt registered [ 1659.840819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1659.861924] alg: No test for adler32 (adler32-zlib) [ 1660.850636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.935659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.956823] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1661.846663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1661.880713] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 1663.017290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.39@tcp1 --nid 192.168.202.39@tcp2 [ 1668.182616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp1 192.168.202.39@tcp2 [ 1668.210121] LNetError: 76592:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.39@tcp1 to 192.168.202.39@tcp2 - Route table may be misconfigured [ 1668.247596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp2 192.168.202.39@tcp1 [ 1675.460987] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:52:56 (1788792776) [ 1675.607713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.646979] LNet: 77161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.655257] LNetError: 77161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.667082] LNet: Removed LNI 192.168.202.39@tcp1 [ 1676.728374] LNet: Removed LNI 192.168.202.39@tcp2 [ 1676.790950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.917048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.955747] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1678.195648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1678.221607] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1679.554650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 1679.616313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.39@tcp [ 1679.671811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp1 192.168.202.39@tcp1 [ 1679.770577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.39@tcp -d 192.168.202.39@tcp -e network_timeout [ 1679.851889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -e network_timeout [ 1679.938337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -e network_timeout [ 1679.988775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -e network_timeout [ 1680.027405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1680.047330] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1680.066353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1680.230314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1680.380413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1680.992403] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1687.345921] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:53:08 (1788792788) [ 1687.506245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.539841] LNet: 78024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.567985] LNetError: 78024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.582309] LNetError: 78024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1687.597503] LNet: Removed LNI 192.168.202.39@tcp [ 1688.686957] LNet: Removed LNI 192.168.202.39@tcp1 [ 1688.764290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.951650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1689.470792] LNet: 78071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.509164] Key type .llcrypt unregistered [ 1690.519700] Key type ._llcrypt unregistered [ 1703.237066] Key type ._llcrypt registered [ 1703.238550] Key type .llcrypt registered [ 1704.620694] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:53:25 (1788792805) [ 1705.009197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.016861] alg: No test for adler32 (adler32-zlib) [ 1705.993236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.629041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1709.429510] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:53:30 (1788792810) [ 1709.729551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.774715] LNet: 79172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.798604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.872419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.904530] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1710.601472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.616255] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1711.577751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1711.623811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp1 [ 1711.678964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp1 [ 1718.527765] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:53:39 (1788792819) [ 1720.494781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg239-server [ 1722.119896] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:53:43 (1788792823) [ 1723.731295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg239-server [ 1725.596700] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:53:46 (1788792826) [ 1727.533698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg239-server [ 1729.281243] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:53:50 (1788792830) [ 1731.010428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg239-server [ 1732.581283] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:53:53 (1788792833) [ 1734.668743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg239-server [ 1737.173921] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:53:57 (1788792837) [ 1738.955385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg239-server [ 1741.067731] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:54:02 (1788792842) [ 1742.891753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg239-server [ 1744.373417] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:54:05 (1788792845) [ 1746.048582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg239-server [ 1747.662376] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:54:08 (1788792848) [ 1749.313482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg239-server [ 1751.007527] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:54:12 (1788792852) [ 1751.190143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.239759] LNet: 83050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.244267] LNetError: 83050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.248462] LNet: Removed LNI 192.168.202.39@tcp [ 1752.293554] LNet: Removed LNI 192.168.202.39@tcp1 [ 1752.357160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.423841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.443489] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1753.329910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1753.425956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1753.690243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.712638] LNet: 83198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.718617] LNetError: 83198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.722191] LNetError: 83198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1754.792796] LNet: Removed LNI 192.168.202.39@tcp [ 1754.849750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.935634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1754.971928] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1755.803174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1755.977243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.004883] LNet: 83343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.034656] LNetError: 83343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.079897] LNet: Removed LNI 192.168.202.39@tcp [ 1756.141848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.239899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.288065] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1757.016402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1757.083634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1757.248934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.317899] LNet: Removed LNI 192.168.202.39@tcp [ 1758.366217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.444948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1758.485109] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1759.289296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1759.424648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.437254] LNetError: 83635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.442158] LNetError: 83635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1760.490744] LNet: Removed LNI 192.168.202.39@tcp [ 1760.537158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.590830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.606463] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1761.471393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1761.602474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1761.902132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.928367] LNet: 83783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.934295] LNet: 83783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1762.982846] LNet: Removed LNI 192.168.202.39@tcp [ 1763.036456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.101579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1763.146330] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1763.840916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1764.117461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.169490] LNetError: 83928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.180233] LNetError: 83928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1764.205872] LNet: Removed LNI 192.168.202.39@tcp [ 1764.240947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.341972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.401117] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1765.725995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1765.917792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1767.404949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.493899] LNet: Removed LNI 192.168.202.39@tcp [ 1768.561962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.703990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1768.760561] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1769.644336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1771.057391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.081719] LNet: 84240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.086554] LNet: 84240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1772.135573] LNet: Removed LNI 192.168.202.39@tcp [ 1772.180585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.238753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.266432] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1772.957945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1773.068111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1773.342593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.378634] LNetError: 84387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.386496] LNetError: 84387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1774.440772] LNet: Removed LNI 192.168.202.39@tcp [ 1774.497725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.657813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1774.705065] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1775.621435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1775.971969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.063390] LNet: Removed LNI 192.168.202.39@tcp [ 1777.106134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.194451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.218069] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1778.049839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1778.137788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1778.388491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.495153] LNet: Removed LNI 192.168.202.39@tcp [ 1779.547158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.629570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1779.691305] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1780.806810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 1781.138299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.187274] LNet: Removed LNI 192.168.202.39@tcp [ 1782.230563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.287866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.306470] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1782.887749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1782.933512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.967322] LNet: Removed LNI 192.168.202.39@tcp [ 1782.998340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.060590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.085228] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1784.030187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1784.126712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.165274] LNet: Removed LNI 192.168.202.39@tcp [ 1784.185692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.283445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1784.414083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.461859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.544294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1784.666203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.722802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.780243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1784.872495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.929821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.992715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1785.025103] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1786.011321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.035547] LNet: Removed LNI 192.168.202.39@tcp [ 1786.049356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.101382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1792.713113] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:54:54 (1788792894) [ 1792.984221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.030260] LNet: 85847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.035254] LNet: 85847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1793.052899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.141286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1793.165321] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/180] [ 1794.519180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.563543] LNetError: 85985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.568791] LNetError: 85985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1794.583854] LNet: Removed LNI 192.168.202.39@tcp231 [ 1794.633554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.728067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1794.764391] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/180] [ 1794.771789] LNet: 85993:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1795.050625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1795.076958] LNet: Removed LNI 192.168.202.39@tcp231 [ 1795.094352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1795.128601] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/180] [ 1796.109915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.147771] LNet: Removed LNI 192.168.202.39@tcp231 [ 1796.190993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.315404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1796.365977] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/0] [ 1796.378761] LNet: 86145:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.919975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1796.978970] LNet: Removed LNI 192.168.202.39@tcp231 [ 1796.999499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1797.060352] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/0] [ 1798.680672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.710337] LNet: Removed LNI 192.168.202.39@tcp231 [ 1798.739654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.812695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1798.838721] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/60] [ 1798.844899] LNet: 86297:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1799.421287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1799.452690] LNet: Removed LNI 192.168.202.39@tcp231 [ 1799.487594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1799.540498] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/60] [ 1801.227663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.285648] LNet: Removed LNI 192.168.202.39@tcp231 [ 1801.344287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.455958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1801.477358] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/180] [ 1801.481933] LNet: 86448:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.774282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1801.801113] LNet: Removed LNI 192.168.202.39@tcp231 [ 1801.814920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1801.838461] LNet: Added LNI 192.168.202.39@tcp231 [8/256/0/180] [ 1811.968437] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:55:12 (1788792912) [ 1812.166982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.235337] LNet: Removed LNI 192.168.202.39@tcp231 [ 1812.272696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.420503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1812.461928] LNet: Added LNI 192.168.202.39@tcp232 [8/256/0/180] [ 1813.818076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1813.920942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1813.966725] LNet: Removed LNI 192.168.202.39@tcp232 [ 1813.988588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1814.030914] LNet: Added LNI 192.168.202.39@tcp232 [8/256/0/180] [ 1815.075487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1815.140285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1815.172922] LNet: Removed LNI 192.168.202.39@tcp232 [ 1815.194289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1815.249276] LNet: Added LNI 192.168.202.39@tcp232 [8/256/0/180] [ 1816.743552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1816.876901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1816.982694] LNet: Removed LNI 192.168.202.39@tcp232 [ 1817.021740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1817.111344] LNet: Added LNI 192.168.202.39@tcp232 [8/256/0/180] [ 1818.398324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1818.493715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1818.574512] LNet: Removed LNI 192.168.202.39@tcp232 [ 1818.607888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1825.539794] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:55:26 (1788792926) [ 1825.899977] LNet: 88289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.903664] LNet: 88289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1826.555773] Key type .llcrypt unregistered [ 1826.558405] Key type ._llcrypt unregistered [ 1832.872376] Key type ._llcrypt registered [ 1832.874670] Key type .llcrypt registered [ 1833.196243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1833.237712] alg: No test for adler32 (adler32-zlib) [ 1834.174251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1834.231100] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1841.861314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1852.424441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.479357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1852.493844] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1853.719794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_interrupt -r 2 [ 1853.755355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_interrupt -r 2 [ 1853.778793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_interrupt -r 2 [ 1853.808848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_interrupt -r 2 [ 1853.843792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_interrupt -r 2 [ 1853.876581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_interrupt -r 2 [ 1853.914694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_interrupt -r 2 [ 1853.979952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_interrupt -r 2 [ 1854.023543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1854.036889] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1854.046874] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1854.304853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1854.348399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1854.373428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1854.432903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1854.481118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1854.550440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1854.610440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1854.662565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1854.700957] LNet: 88956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1854.733703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1854.785685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1854.839737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1854.883687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1854.924724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1854.960799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1855.961412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_dropped -r 2 [ 1856.030732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_dropped -r 2 [ 1856.094171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_dropped -r 2 [ 1856.157106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_dropped -r 2 [ 1856.250965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_dropped -r 2 [ 1856.324428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_dropped -r 2 [ 1856.400016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_dropped -r 2 [ 1856.476490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_dropped -r 2 [ 1856.528226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1856.581581] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1856.982685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.023491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1857.052635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.077788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.117022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.186155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.222532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.247902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.260970] LNet: 89077:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.281134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.315844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.364285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.424490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.467740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1857.543848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1858.305282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_aborted -r 2 [ 1858.331928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_aborted -r 2 [ 1858.355260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_aborted -r 2 [ 1858.381746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_aborted -r 2 [ 1858.405446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_aborted -r 2 [ 1858.428261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_aborted -r 2 [ 1858.455120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_aborted -r 2 [ 1858.479703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_aborted -r 2 [ 1858.512486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1858.528265] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1858.743480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1858.770676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1858.795938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.839342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.895466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.932234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1858.989698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.052658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.069111] LNet: 89197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.084588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1859.114754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1859.140406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1859.165159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1859.191944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1859.337876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1860.070699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_no_route -r 2 [ 1860.126898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_no_route -r 2 [ 1860.181507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_no_route -r 2 [ 1860.224315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_no_route -r 2 [ 1860.262662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_no_route -r 2 [ 1860.334782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_no_route -r 2 [ 1860.392714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_no_route -r 2 [ 1860.465188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_no_route -r 2 [ 1860.530768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1860.551857] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1860.842622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1860.870381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1860.904098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1860.937057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1860.974296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1861.010876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1861.043254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1861.076716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1861.100945] LNet: 89317:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1861.123631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1861.157151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1861.204231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1861.254094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1861.311609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1861.409355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1862.245019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e local_timeout -r 2 [ 1862.296768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e local_timeout -r 2 [ 1862.368394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e local_timeout -r 2 [ 1862.410884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e local_timeout -r 2 [ 1862.464599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e local_timeout -r 2 [ 1862.496769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e local_timeout -r 2 [ 1862.522974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e local_timeout -r 2 [ 1862.555912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e local_timeout -r 2 [ 1862.598593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1862.912563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1862.937781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1862.964993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1862.994369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1863.022332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1863.064546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1863.092336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1863.126120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1863.154995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1863.183969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1863.215072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1863.237982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1863.276092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1863.306074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.009700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp -m GET -e remote_dropped -r 2 [ 1864.109856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.139@tcp1 -m GET -e remote_dropped -r 2 [ 1864.177497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp -m GET -e remote_dropped -r 2 [ 1864.233592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp -d 192.168.202.39@tcp1 -m GET -e remote_dropped -r 2 [ 1864.350964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp -m GET -e remote_dropped -r 2 [ 1864.427994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.139@tcp1 -m GET -e remote_dropped -r 2 [ 1864.511991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp -m GET -e remote_dropped -r 2 [ 1864.574520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.39@tcp1 -d 192.168.202.39@tcp1 -m GET -e remote_dropped -r 2 [ 1864.626471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.39@tcp 192.168.202.139@tcp [ 1864.646669] LNet: There was an unexpected network error while writing to 192.168.202.139: rc = -22 [ 1864.651342] LNet: Skipped 1 previous similar message [ 1864.654035] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.139@tcp [ 1864.935968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1864.978528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1865.014608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1865.045545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1865.077663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1865.099881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1865.133076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.152638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1865.165505] LNet: 89557:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1865.169741] LNet: 89557:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1865.184598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1865.205756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1865.234287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1865.261763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1865.282644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1865.302855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1871.273387] LNet: 89636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.277460] LNetError: 89636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.285538] LNet: Removed LNI 192.168.202.39@tcp [ 1871.293800] LNet: Removed LNI 192.168.202.39@tcp1 [ 1871.591718] Key type .llcrypt unregistered [ 1871.593905] Key type ._llcrypt unregistered [ 1888.479486] Key type ._llcrypt registered [ 1888.481490] Key type .llcrypt registered [ 1889.502504] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:56:30 (1788792990) [ 1890.173954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1890.200735] alg: No test for adler32 (adler32-zlib) [ 1891.095598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.164558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.188455] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1892.031787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1892.051539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.39@tcp [ 1892.063776] LNetError: 90617:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.39@tcp to recovery [ 1892.146758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.39@tcp [ 1892.246760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1898.113816] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:56:39 (1788792999) [ 1898.379177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.420595] LNet: 91208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.429866] LNetError: 91208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.449825] LNet: Removed LNI 192.168.202.39@tcp [ 1898.470688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.576901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.628155] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1899.476972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1899.507798] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 1900.690647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.39@tcp [ 1900.720436] LNetError: 91474:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.39@tcp to recovery [ 1901.600275] LNet: 1 local NIs in recovery (showing 1): 192.168.202.39@tcp [ 1901.616725] LNet: There was an unexpected network error while writing to 192.168.202.39: rc = -22 [ 1901.963577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.39@tcp [ 1909.876964] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:56:51 (1788793011) [ 1911.622208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg239-server [ 1913.254503] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:56:54 (1788793014) [ 1914.782940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg239-server [ 1916.564449] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:56:57 (1788793017) [ 1918.226252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg239-server [ 1919.533988] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:57:01 (1788793021) [ 1919.757041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.801267] LNet: 93103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.811721] LNetError: 93103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.822842] LNet: Removed LNI 192.168.202.39@tcp [ 1919.846353] LNet: Removed LNI 192.168.202.39@tcp1 [ 1919.878580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.953975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.975220] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1920.577926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1920.656413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1920.702204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1920.819850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1920.905166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1921.008427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1927.900747] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:57:08 (1788793028) [ 1928.006697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.050388] LNet: 93828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.056355] LNetError: 93828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.060991] LNetError: 93828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1928.101965] LNet: Removed LNI 192.168.202.39@tcp [ 1928.131993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.223919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.250776] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1934.926598] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:57:16 (1788793036) [ 1935.155848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1935.178201] LNet: 94538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.185340] LNetError: 94538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.202449] LNet: Removed LNI 192.168.202.39@tcp [ 1935.248994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.315198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.332954] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1941.449955] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:57:22 (1788793042) [ 1943.078350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1945.097651] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:57:26 (1788793046) [ 1945.558307] LNet: 95573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.564498] LNetError: 95573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.579057] LNet: Removed LNI 192.168.202.39@tcp [ 1946.103914] Key type .llcrypt unregistered [ 1946.105212] Key type ._llcrypt unregistered [ 1954.129453] Key type ._llcrypt registered [ 1954.131697] Key type .llcrypt registered [ 1954.536230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1954.569412] alg: No test for adler32 (adler32-zlib) [ 1955.451298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1955.496985] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 1965.743919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 1972.655812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.139@tcp [ 1973.341534] LNet: 96000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.362229] LNetError: 96000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.378951] LNet: Removed LNI 192.168.202.39@tcp [ 1973.934588] Key type .llcrypt unregistered [ 1973.939267] Key type ._llcrypt unregistered [ 1986.679696] Key type ._llcrypt registered [ 1986.684890] Key type .llcrypt registered [ 1988.103428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1989.726885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1991.909229] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:58:12 (1788793092) [ 1993.857715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1995.322654] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:58:16 (1788793096) [ 1997.592959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg239-server [ 1999.344366] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:58:20 (1788793100) [ 2001.355422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg239-server [ 2002.991581] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:58:24 (1788793104) [ 2004.123462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.250121] alg: No test for adler32 (adler32-zlib) [ 2005.343254] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2005.608323] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2007.367224] Key type lgssc registered [ 2008.882468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2027.771825] Key type lgssc unregistered [ 2028.040734] LNet: 99179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.044746] LNetError: 99179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.057602] LNet: Removed LNI 192.168.202.39@tcp [ 2028.754502] Key type .llcrypt unregistered [ 2028.756250] Key type ._llcrypt unregistered [ 2042.448526] Key type ._llcrypt registered [ 2042.450509] Key type .llcrypt registered [ 2043.792698] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:59:04 (1788793144) [ 2044.332449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.343921] alg: No test for adler32 (adler32-zlib) [ 2045.290214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.345709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.380113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.425307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.465533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2052.010385] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:59:13 (1788793153) [ 2052.282619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.315262] LNet: 100595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.350498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.405793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.449371] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2053.809840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.850606] LNet: 100739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.854852] LNetError: 100739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.883045] LNet: Removed LNI 192.168.202.39@tcp [ 2053.901781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.969007] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2054.001619] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2054.573468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.590400] LNetError: 100767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.598732] LNet: Removed LNI 192.168.202.39@tcp [ 2054.643410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.740624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2054.810419] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2063.002315] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:59:23 (1788793163) [ 2063.156353] LNet: 101349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.164109] LNet: 101349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2063.188878] LNetError: 101349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.232077] LNet: Removed LNI 192.168.202.39@tcp [ 2064.093601] Key type .llcrypt unregistered [ 2064.097969] Key type ._llcrypt unregistered [ 2071.230905] Key type ._llcrypt registered [ 2071.236631] Key type .llcrypt registered [ 2071.457252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2071.471941] alg: No test for adler32 (adler32-zlib) [ 2072.432455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.558710] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2072.564257] LNet: *** cfs_fail_loc=e003, val=100*** [ 2073.318900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2073.333600] LNet: *** cfs_fail_loc=e003, val=100*** [ 2073.336971] LNet: Skipped 2 previous similar messages [ 2073.342798] Ping reply received with mapping: [ 2073.346229] found metadata reply NI[192.168.202.39@tcp] [ 2073.349664] 27 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2073.356518] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2073.358583] 5a 5a 5a 5a 5a 5a 5a 5a [ 2073.474821] LNet: 101760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.482344] LNetError: 101760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.495932] LNet: Removed LNI 192.168.202.39@tcp [ 2074.131412] Key type .llcrypt unregistered [ 2074.133556] Key type ._llcrypt unregistered [ 2086.622451] Key type ._llcrypt registered [ 2086.626697] Key type .llcrypt registered [ 2088.703053] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:59:48 (1788793188) [ 2089.360659] Key type .llcrypt unregistered [ 2089.368251] Key type ._llcrypt unregistered [ 2095.258159] Key type ._llcrypt registered [ 2095.259704] Key type .llcrypt registered [ 2095.444307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2095.459599] alg: No test for adler32 (adler32-zlib) [ 2096.523460] LNetError: 102819:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2096.865420] Key type .llcrypt unregistered [ 2096.875427] Key type ._llcrypt unregistered [ 2101.053212] Key type ._llcrypt registered [ 2101.055275] Key type .llcrypt registered [ 2102.380690] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:00:03 (1788793203) [ 2102.841537] Key type .llcrypt unregistered [ 2102.846893] Key type ._llcrypt unregistered [ 2108.178089] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2108.182205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2110.758454] Key type ._llcrypt registered [ 2110.763446] Key type .llcrypt registered [ 2110.996138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2111.019124] alg: No test for adler32 (adler32-zlib) [ 2111.993673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.050755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2122.727791] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:00:23 (1788793223) [ 2123.002194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.050098] LNet: 104607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.063535] LNetError: 104607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2123.103455] LNet: Removed LNI 10.1.2.3@tcp [ 2123.132078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.017450] Key type .llcrypt unregistered [ 2124.019699] Key type ._llcrypt unregistered [ 2131.937707] Key type ._llcrypt registered [ 2131.939306] Key type .llcrypt registered [ 2132.241419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2132.268194] alg: No test for adler32 (adler32-zlib) [ 2133.264555] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2133.371588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2133.530975] LNet: 104901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.535006] LNetError: 104901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.564791] LNet: Removed LNI 192.168.202.39@tcp [ 2135.329672] Key type .llcrypt unregistered [ 2135.337210] Key type ._llcrypt unregistered [ 2148.498910] Key type ._llcrypt registered [ 2148.502908] Key type .llcrypt registered [ 2149.863647] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:00:50 (1788793250) [ 2150.962612] Key type .llcrypt unregistered [ 2150.967718] Key type ._llcrypt unregistered [ 2157.633883] Key type ._llcrypt registered [ 2157.637371] Key type .llcrypt registered [ 2157.957763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2157.988321] alg: No test for adler32 (adler32-zlib) [ 2165.333478] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:01:06 (1788793266) [ 2165.758142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.801295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.891655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.919851] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2166.779690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.815753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2166.853523] LNet: Removed LNI 192.168.202.39@tcp [ 2167.230505] LNet: 106670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.617594] Key type .llcrypt unregistered [ 2167.619206] Key type ._llcrypt unregistered [ 2181.133126] Key type ._llcrypt registered [ 2181.134693] Key type .llcrypt registered [ 2182.679981] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:01:23 (1788793283) [ 2183.056720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.072675] alg: No test for adler32 (adler32-zlib) [ 2184.059385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.147505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.196714] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2184.885551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2184.913619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.39@tcp [ 2184.927450] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2184.931386] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2184.934340] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2184, 0, 0 [ 2184.938841] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2184.941352] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2184.943799] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2184.946783] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2184, 0, 0 [ 2184.950916] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2184.953935] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2184.960967] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2184.968113] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2184, 0, 2234 [ 2184.975068] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2184.976928] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2184.990881] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2185.004343] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2184, 0, 2234 [ 2185.018357] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2191.202016] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:01:32 (1788793292) [ 2191.657212] LNet: 108211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.668945] LNetError: 108211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.743385] LNet: Removed LNI 192.168.202.39@tcp [ 2193.666505] Key type .llcrypt unregistered [ 2193.668672] Key type ._llcrypt unregistered [ 2201.087754] Key type ._llcrypt registered [ 2201.094104] Key type .llcrypt registered [ 2201.373336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2201.386585] alg: No test for adler32 (adler32-zlib) [ 2202.315771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2202.350177] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2210.450482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 2219.617232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.694631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2219.744805] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 2220.602032] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2220.770192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2220.780258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2220.791525] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2223.175892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2223.205275] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2233.040218] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:02:14 (1788793334) [ 2233.193179] LNet: 109720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.202432] LNetError: 109720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.274674] LNet: Removed LNI 192.168.202.39@tcp [ 2234.278949] LNetError: 109720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.287449] LNet: Removed LNI 192.168.202.39@tcp1 [ 2234.306704] LNet: Removed LNI 10.1.2.3@tcp99 [ 2234.997275] Key type .llcrypt unregistered [ 2234.999205] Key type ._llcrypt unregistered [ 2242.076198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2242.081223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2244.784835] Key type ._llcrypt registered [ 2244.791326] Key type .llcrypt registered [ 2245.036247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2245.047169] alg: No test for adler32 (adler32-zlib) [ 2246.079850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.170800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.204192] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2247.113888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2247.144882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2249.314422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2249.364510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2249.431444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2249.453355] LNetError: 110474:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2249.499778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2249.591689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2250.206426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2250.294772] LNet: 110495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.306700] LNetError: 110495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2250.309135] LNet: Removed LNI 10.1.2.3@tcp [ 2251.365892] LNet: Removed LNI 192.168.202.39@tcp [ 2252.144576] Key type .llcrypt unregistered [ 2252.147768] Key type ._llcrypt unregistered [ 2265.663814] Key type ._llcrypt registered [ 2265.666554] Key type .llcrypt registered [ 2266.773797] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:02:48 (1788793368) [ 2267.369063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.386933] alg: No test for adler32 (adler32-zlib) [ 2268.282927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.352939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.372511] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2269.077912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2269.157775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg239-client@tcp [ 2274.868347] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:02:56 (1788793376) [ 2275.231054] LNet: 112033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.249256] LNetError: 112033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.267184] LNet: Removed LNI 192.168.202.39@tcp [ 2275.992634] Key type .llcrypt unregistered [ 2275.997251] Key type ._llcrypt unregistered [ 2283.064441] Key type ._llcrypt registered [ 2283.066734] Key type .llcrypt registered [ 2283.226851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.243681] alg: No test for adler32 (adler32-zlib) [ 2284.205957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2284.260368] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2284.310241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2290.324990] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:03:11 (1788793391) [ 2292.300525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2293.836626] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:03:15 (1788793395) [ 2295.181724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2296.926228] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:03:18 (1788793398) [ 2297.052664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.093447] LNet: 113453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.105496] LNetError: 113453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.128990] LNet: Removed LNI 192.168.202.39@tcp [ 2297.164092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.259487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.286521] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2298.030815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2298.074510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2304.520991] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:03:26 (1788793406) [ 2304.726657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.741805] LNet: 114169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.746832] LNetError: 114169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.767236] LNet: Removed LNI 192.168.202.39@tcp [ 2305.806950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.875905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2305.915571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2306.169335] LNet: 114209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.491943] Key type .llcrypt unregistered [ 2306.493146] Key type ._llcrypt unregistered [ 2316.917976] Key type ._llcrypt registered [ 2316.921408] Key type .llcrypt registered [ 2318.088247] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:03:39 (1788793419) [ 2318.734570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.751622] alg: No test for adler32 (adler32-zlib) [ 2319.759762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.804597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.830565] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2320.583483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2320.628221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2327.682279] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:03:48 (1788793428) [ 2327.807690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.890288] LNet: 115723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.900438] LNetError: 115723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.936960] LNet: Removed LNI 192.168.202.39@tcp [ 2327.961843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.074904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2328.133942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2334.382579] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:03:55 (1788793435) [ 2334.479098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.514892] LNet: 116304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.542328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.595431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2334.896150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.944634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.046150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2335.450812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.485380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.579244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.605350] LNet: 116343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.651417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.804534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2336.444594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2336.502850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2336.603181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2336.890886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2336.916372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2344.470988] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:04:05 (1788793445) [ 2344.626460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.661399] LNet: 116944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.688309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.754704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.788516] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2345.722533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.746899] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2346.994338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2347.060822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2347.373512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.394242] LNetError: 117223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.447349] LNet: Removed LNI 192.168.202.39@tcp [ 2348.449085] LNetError: 117223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.500383] LNet: Removed LNI 192.168.202.39@tcp2 [ 2348.595838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.768631] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2348.784857] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2348.796235] LNet: 117229:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2349.480266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2349.884958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.916903] LNet: 117267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.925169] LNet: 117267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2350.943286] LNet: Removed LNI 192.168.202.39@tcp [ 2350.948621] LNetError: 117267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.963925] LNetError: 117267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2350.994295] LNet: Removed LNI 192.168.202.39@tcp2 [ 2351.030820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.197057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2351.275764] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2352.911155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2352.961911] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2354.443579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2354.490029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2354.593055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2354.920813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.970062] LNetError: 117551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.999726] LNet: Removed LNI 192.168.202.39@tcp [ 2357.049427] LNet: Removed LNI 192.168.202.39@tcp2 [ 2357.108736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.239710] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2357.263502] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2357.271747] LNet: 117558:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.113734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2358.559906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.603339] LNet: 117595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.607818] LNet: 117595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2358.628959] LNet: Removed LNI 192.168.202.39@tcp [ 2359.662372] LNet: Removed LNI 192.168.202.39@tcp2 [ 2359.699109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.783703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.810280] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2361.080610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.117982] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2362.537793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2362.649417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2363.021444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.065697] LNetError: 117874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.084333] LNetError: 117874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2363.098566] LNet: Removed LNI 192.168.202.39@tcp [ 2363.128129] LNet: Removed LNI 192.168.202.39@tcp2 [ 2363.197068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.313471] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2363.324630] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2363.342900] LNet: 117880:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.045988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2364.247249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.279508] LNet: Removed LNI 192.168.202.39@tcp [ 2365.299595] LNet: Removed LNI 192.168.202.39@tcp2 [ 2365.314786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.381370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.398952] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2366.243787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2366.262607] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2367.382729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2367.407483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2367.452121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2367.632168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.673588] LNet: Removed LNI 192.168.202.39@tcp [ 2369.710885] LNet: Removed LNI 192.168.202.39@tcp2 [ 2369.765534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.838332] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2369.844623] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2369.848256] LNet: 118208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.424472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2370.657974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.707077] LNet: Removed LNI 192.168.202.39@tcp [ 2371.750495] LNet: Removed LNI 192.168.202.39@tcp2 [ 2371.796495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.893672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.926310] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2373.160527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2373.203806] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2374.372746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2374.404918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2374.469944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2374.879821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.897880] LNet: 118542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.901858] LNet: 118542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2375.967173] LNet: Removed LNI 192.168.202.39@tcp [ 2375.979837] LNet: Removed LNI 192.168.202.39@tcp2 [ 2376.049077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.163631] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2376.170036] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2376.175849] LNet: 118548:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.550517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2376.831863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.887474] LNet: Removed LNI 192.168.202.39@tcp [ 2377.905980] LNet: Removed LNI 192.168.202.39@tcp2 [ 2377.966939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.055113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.092513] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2378.805876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2378.821994] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2379.477472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2379.512531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2379.534472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2379.580445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2379.855559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.878376] LNetError: 118902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.883937] LNetError: 118902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 11 previous similar messages [ 2379.890294] LNet: Removed LNI 192.168.202.39@tcp [ 2380.968230] LNet: Removed LNI 192.168.202.39@tcp2 [ 2381.019605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.089391] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2381.097845] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2381.595279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2388.079314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2389.892854] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:04:51 (1788793491) [ 2390.164965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.196907] LNet: Removed LNI 192.168.202.39@tcp [ 2391.271321] LNet: Removed LNI 192.168.202.39@tcp2 [ 2391.338888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.905873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.946068] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2393.045540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2393.085476] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2394.390573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.446509] LNet: Removed LNI 10.1.2.3@tcp [ 2394.484264] LNet: Removed LNI 192.168.202.39@tcp [ 2394.536763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.616083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2394.684653] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2394.687966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2395.423227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2395.458722] LNet: Removed LNI 192.168.202.39@tcp [ 2395.472263] LNet: Removed LNI 10.1.2.3@tcp [ 2402.611270] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:05:03 (1788793503) [ 2402.741857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.802412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.867417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.884936] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2403.681739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2403.714236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.753548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.801457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.836977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.882696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.940275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2403.992380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2404.021502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2404.055527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2404.103702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.39@tcp [ 2404.301675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2411.326789] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:05:12 (1788793512) [ 2411.494676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.542694] LNet: 121277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.550252] LNet: 121277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2412.586348] LNet: Removed LNI 192.168.202.39@tcp [ 2412.631912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.722065] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2412.893142] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2412.899296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2412.903986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2415.315215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.345229] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2416.028364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2416.050693] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2418.114956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2418.199686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2418.305599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.39@tcp -m PUT -e local_timeout -r 1 [ 2418.396384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2418.483304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2418.491330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2418.963859] LNetError: 121349:0:(lib-move.c:4801:lnet_parse()) 192.168.202.39@tcp, src 192.168.202.39@tcp: Dropping PUT (error -108 looking up sender) [ 2420.001078] LNetError: 121812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2420.017559] LNetError: 121812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2420.033582] LNet: Removed LNI 10.1.2.3@tcp [ 2421.094571] LNet: Removed LNI 192.168.202.39@tcp [ 2421.680956] Key type .llcrypt unregistered [ 2421.683691] Key type ._llcrypt unregistered [ 2434.813199] Key type ._llcrypt registered [ 2434.815372] Key type .llcrypt registered [ 2436.548890] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:05:37 (1788793537) [ 2437.263917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.280460] alg: No test for adler32 (adler32-zlib) [ 2438.312790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.716227] LNet: 122670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.596352] Key type .llcrypt unregistered [ 2439.600613] Key type ._llcrypt unregistered [ 2448.889071] Key type ._llcrypt registered [ 2448.891977] Key type .llcrypt registered [ 2449.135179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2449.155546] alg: No test for adler32 (adler32-zlib) [ 2450.095755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2450.114257] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2458.472988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 2458.751964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2459.329247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.139@tcp [ 2459.359182] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.139@tcp [ 2459.486491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2459.507242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2459.532487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2459.555466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2459.582248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2459.628236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2459.667774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2459.704923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2459.724398] LNet: 123125:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2459.740874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2459.763947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2459.793809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2459.821990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2459.851633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2459.878271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2460.445533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2460.651810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.683869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2460.727192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.754986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.783912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.805970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2460.821679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2460.844874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2460.862841] LNet: 123242:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2460.876757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2460.905251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2460.931280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2460.957272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2460.983244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2461.015420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2461.859141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.139@tcp [ 2462.144687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2462.184788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2462.216747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2462.260375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2462.304307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2462.328718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2462.345991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2462.368660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2462.408396] LNet: 123359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2462.467398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2462.531122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2462.568500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2462.596041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2462.640385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2462.684405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2463.298442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2463.511557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2463.561076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2463.591790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2463.612949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2463.638758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2463.701702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2463.760665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2463.807782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2463.854247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2463.894695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2463.932416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2463.979880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2464.029186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2464.084718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2464.349159] LNet: 123526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.359288] LNetError: 123526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.375824] LNet: Removed LNI 192.168.202.39@tcp [ 2464.818960] Key type .llcrypt unregistered [ 2464.821275] Key type ._llcrypt unregistered [ 2480.188732] Key type ._llcrypt registered [ 2480.192760] Key type .llcrypt registered [ 2481.473832] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:06:22 (1788793582) [ 2482.160633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2482.186581] alg: No test for adler32 (adler32-zlib) [ 2483.142447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.594508] LNet: 124415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.315788] Key type .llcrypt unregistered [ 2484.318684] Key type ._llcrypt unregistered [ 2491.491591] Key type ._llcrypt registered [ 2491.493649] Key type .llcrypt registered [ 2491.665305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2491.675665] alg: No test for adler32 (adler32-zlib) [ 2492.604868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2492.655238] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2499.993869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.139@tcp [ 2509.833410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.864428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2509.885148] LNet: Added LNI 192.168.202.39@tcp1 [8/256/0/180] [ 2511.615821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2512.223859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.139@tcp [ 2512.486752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2512.518436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2512.581375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2512.610357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2512.650410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2512.677785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2512.713758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2512.745667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2512.761717] LNet: 125072:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2512.776414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2512.814964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2512.840810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2512.865319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2512.894913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2512.920074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2513.685935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2513.974143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2514.022760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2514.066271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2514.110102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2514.144625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2514.211430] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.139@tcp, 192.168.202.139@tcp1 [ 2514.238983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2514.300370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2514.349936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2514.387823] LNet: 125189:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2514.432505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2514.474456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2514.517785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2514.562308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2514.601446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2514.653394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2515.542526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.139@tcp [ 2515.817698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2515.862498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2515.896627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2515.939200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2515.971873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2516.012200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2516.050895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2516.096565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2516.111668] LNet: 125307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2516.135835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2516.167415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2516.216525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2516.264039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2516.310162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2516.387467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2517.387667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2517.785236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2517.849779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2517.901403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2517.967913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2518.003767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2518.037996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2518.068706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2518.110170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2518.150448] LNet: 125424:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2518.166874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2518.210496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2518.269279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2518.311640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2518.348025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2518.401914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2523.821612] LNet: 125509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.832340] LNetError: 125509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.843385] LNet: Removed LNI 192.168.202.39@tcp [ 2523.861928] LNet: Removed LNI 192.168.202.39@tcp1 [ 2524.488889] Key type .llcrypt unregistered [ 2524.492509] Key type ._llcrypt unregistered [ 2541.481540] Key type ._llcrypt registered [ 2541.490221] Key type .llcrypt registered [ 2543.391845] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:07:23 (1788793643) [ 2543.962890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.973891] alg: No test for adler32 (adler32-zlib) [ 2545.052620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.149927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2545.206878] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2546.106887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2546.128670] LNet: Added LNI 192.168.202.39@tcp2 [8/256/0/180] [ 2546.979654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.39@tcp [ 2547.040487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2548.063337] LNet: Removed LNI 192.168.202.39@tcp [ 2554.981948] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:07:36 (1788793656) [ 2557.027959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2559.063848] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:07:39 (1788793659) [ 2567.108543] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:07:48 (1788793668) [ 2574.858419] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:07:56 (1788793676) [ 2583.113886] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:08:04 (1788793684) [ 2590.114335] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:08:11 (1788793691) [ 2597.119468] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:08:18 (1788793698) [ 2605.093777] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:08:26 (1788793706) [ 2612.458788] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:08:33 (1788793713) [ 2619.047473] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:08:40 (1788793720) [ 2625.735603] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:08:46 (1788793726) [ 2633.118628] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:08:53 (1788793733) [ 2641.689887] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:09:02 (1788793742) [ 2649.321536] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:09:10 (1788793750) [ 2656.317712] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:09:17 (1788793757) [ 2662.825822] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:09:24 (1788793764) [ 2670.345292] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:09:31 (1788793771) [ 2677.688737] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:09:38 (1788793778) [ 2684.720668] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:09:46 (1788793786) [ 2684.837028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2684.884283] LNet: 137198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.893759] LNetError: 137198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2684.921100] LNet: Removed LNI 192.168.202.39@tcp2 [ 2684.965682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2685.107552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2685.146838] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2686.245132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2686.298977] LNet: 137345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.309951] LNetError: 137345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.328198] LNet: Removed LNI 192.168.202.39@tcp [ 2686.350310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2686.410699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2686.462601] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2693.297580] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:09:54 (1788793794) [ 2702.449704] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:10:03 (1788793803) [ 2710.063027] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:10:11 (1788793811) [ 2716.913300] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:10:18 (1788793818) [ 2723.910671] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:10:25 (1788793825) [ 2731.837267] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:10:32 (1788793832) [ 2739.705986] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:10:40 (1788793840) [ 2747.344133] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:10:48 (1788793848) [ 2757.146566] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:10:57 (1788793857) [ 2767.171291] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:11:07 (1788793867) [ 2774.749221] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:11:15 (1788793875) [ 2781.389155] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:11:23 (1788793883) [ 2789.497184] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2521 sec ========== 11:11:30 (1788793890) [ 2789.923912] LNet: 144813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.929985] LNetError: 144813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.947909] LNet: Removed LNI 192.168.202.39@tcp [ 2790.881834] Key type .llcrypt unregistered [ 2790.884099] Key type ._llcrypt unregistered [ 2808.685859] Key type ._llcrypt registered [ 2808.690467] Key type .llcrypt registered [ 2809.413527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2809.445183] alg: No test for adler32 (adler32-zlib) [ 2810.866465] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2811.233381] LNet: Added LNI 192.168.202.39@tcp [8/256/0/180] [ 2812.959179] Key type lgssc registered [ 2813.839192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.685937] Lustre: Mounted lustre-client - version 2.17.57_82_gfdb2c0c [ 2884.960286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2894.281077] Lustre: Unmounted lustre-client [ 2919.071900] Key type lgssc unregistered [ 2919.538183] LNet: 147773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.547829] LNetError: 147773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.569434] LNet: Removed LNI 192.168.202.39@tcp [ 2920.835622] Key type .llcrypt unregistered [ 2920.842836] Key type ._llcrypt unregistered