[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468751003 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.008750] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012078] LSM: Security Framework initializing [ 0.013000] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014000] *** VALIDATE selinux *** [ 0.022499] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027619] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030085] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031000] *** VALIDATE tmpfs *** [ 0.031423] *** VALIDATE proc *** [ 0.033074] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.041684] debug: unmapping init [mem 0xffffffffb1659000-0xffffffffb1660fff] [ 0.043261] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049021] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052291] rcu: Hierarchical SRCU implementation. [ 0.054952] smp: Bringing up secondary CPUs ... [ 0.055622] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.063082] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.116015] node 0 deferred pages initialised in 43ms [ 0.121011] devtmpfs: initialized [ 0.122000] x86/mm: Memory block size: 128MB [ 0.126000] gcov: version magic: 0x41383552 [ 0.131396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132175] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133671] pinctrl core: initialized pinctrl subsystem [ 0.134293] [ 0.135000] ************************************************************* [ 0.136014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137016] ** ** [ 0.138012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139010] ** ** [ 0.140013] ** This means that this kernel is built to expose internal ** [ 0.141010] ** IOMMU data structures, which may compromise security on ** [ 0.142013] ** your system. ** [ 0.143013] ** ** [ 0.144013] ** If you see this message and you are not debugging the ** [ 0.145011] ** kernel, report this immediately to your vendor! ** [ 0.146013] ** ** [ 0.147013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148014] ************************************************************* [ 0.149708] NET: Registered protocol family 16 [ 0.150499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.151061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.152062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.153617] cpuidle: using governor menu [ 0.155953] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.158596] PCI: Using configuration type 1 for base access [ 0.162181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170401] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171177] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.172685] cryptd: max_cpu_qlen set to 1000 [ 0.173322] ACPI: Added _OSI(Module Device) [ 0.175013] ACPI: Added _OSI(Processor Device) [ 0.176009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.178012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.183958] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.190926] ACPI: Interpreter enabled [ 0.191058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.192009] ACPI: Using IOAPIC for interrupt routing [ 0.193092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.194667] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.206851] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.208017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.212096] acpiphp: Slot [2] registered [ 0.213133] acpiphp: Slot [5] registered [ 0.214085] acpiphp: Slot [6] registered [ 0.215146] acpiphp: Slot [3] registered [ 0.216119] acpiphp: Slot [4] registered [ 0.217098] acpiphp: Slot [7] registered [ 0.218083] acpiphp: Slot [8] registered [ 0.219084] acpiphp: Slot [9] registered [ 0.220098] acpiphp: Slot [10] registered [ 0.221181] acpiphp: Slot [11] registered [ 0.222094] acpiphp: Slot [12] registered [ 0.223094] acpiphp: Slot [13] registered [ 0.224109] acpiphp: Slot [14] registered [ 0.225181] acpiphp: Slot [15] registered [ 0.226194] acpiphp: Slot [16] registered [ 0.227121] acpiphp: Slot [17] registered [ 0.228190] acpiphp: Slot [18] registered [ 0.229152] acpiphp: Slot [19] registered [ 0.230126] acpiphp: Slot [20] registered [ 0.231083] acpiphp: Slot [21] registered [ 0.232067] acpiphp: Slot [22] registered [ 0.233109] acpiphp: Slot [23] registered [ 0.234105] acpiphp: Slot [24] registered [ 0.235148] acpiphp: Slot [25] registered [ 0.236085] acpiphp: Slot [26] registered [ 0.238095] acpiphp: Slot [27] registered [ 0.239213] acpiphp: Slot [28] registered [ 0.240135] acpiphp: Slot [29] registered [ 0.241118] acpiphp: Slot [30] registered [ 0.242334] acpiphp: Slot [31] registered [ 0.243045] PCI host bridge to bus 0000:00 [ 0.244019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.247020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.248018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.249020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.250219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252253] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.253823] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.274787] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.276020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.279018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.285016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.289532] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295083] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.304115] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.309142] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.331014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341060] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.349210] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.400523] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.420014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.450015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468703] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.470475] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.474410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.476556] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.479203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.484075] iommu: Default domain type: Passthrough [ 0.486276] SCSI subsystem initialized [ 0.488286] ACPI: bus type USB registered [ 0.489165] usbcore: registered new interface driver usbfs [ 0.491652] usbcore: registered new interface driver hub [ 0.493074] usbcore: registered new device driver usb [ 0.495427] pps_core: LinuxPPS API ver. 1 registered [ 0.498011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.500049] PTP clock support registered [ 0.502166] EDAC MC: Ver: 3.0.0 [ 0.503363] PCI: Using ACPI for IRQ routing [ 0.504640] NetLabel: Initializing [ 0.505000] NetLabel: domain hash size = 128 [ 0.506007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.508156] NetLabel: unlabeled traffic allowed by default [ 0.509374] vgaarb: loaded [ 0.510324] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.512010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.518385] clocksource: Switched to clocksource kvm-clock [ 0.796968] VFS: Disk quotas dquot_6.6.0 [ 0.800261] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.811296] *** VALIDATE ramfs *** [ 0.816231] *** VALIDATE hugetlbfs *** [ 0.819129] pnp: PnP ACPI init [ 0.823178] pnp: PnP ACPI: found 6 devices [ 0.920187] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.936432] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.949929] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956288] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.968738] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.978045] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.987712] NET: Registered protocol family 2 [ 0.997703] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.025827] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.050180] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.084729] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.107385] TCP: Hash tables configured (established 65536 bind 65536) [ 1.123134] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.135276] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.143870] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.162752] NET: Registered protocol family 1 [ 1.194572] RPC: Registered named UNIX socket transport module. [ 1.196678] RPC: Registered udp transport module. [ 1.206832] RPC: Registered tcp transport module. [ 1.217039] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.234361] NET: Registered protocol family 44 [ 1.242562] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.246658] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.254254] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.263709] PCI: CLS 0 bytes, default 64 [ 1.265302] Unpacking initramfs... [ 4.605548] debug: unmapping init [mem 0xffff952bbcc64000-0xffff952bbffcffff] [ 4.627669] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.633269] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.639319] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.550669] Initialise system trusted keyrings [ 5.551849] Key type blacklist registered [ 5.553402] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.561545] zbud: loaded [ 5.566909] *** VALIDATE nfs *** [ 5.568385] *** VALIDATE nfs4 *** [ 5.571656] pstore: using deflate compression [ 5.576689] Platform Keyring initialized [ 5.721694] NET: Registered protocol family 38 [ 5.723289] Key type asymmetric registered [ 5.725261] Asymmetric key parser 'x509' registered [ 5.727337] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.730861] io scheduler mq-deadline registered [ 5.733319] io scheduler kyber registered [ 5.735489] io scheduler bfq registered [ 5.737155] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.740561] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.743774] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.753255] ACPI: Power Button [PWRF] [ 5.764250] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.778520] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.798282] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.833236] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.883664] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.907352] Non-volatile memory driver v1.3 [ 5.915121] Linux agpgart interface v0.103 [ 6.006886] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 6.011752] vda: detected capacity change from 0 to 76693504 [ 6.052782] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.057788] vdb: detected capacity change from 0 to 1073741824 [ 6.072553] libphy: Fixed MDIO Bus: probed [ 6.096366] usbcore: registered new interface driver usbserial_generic [ 6.100339] usbserial: USB Serial support registered for generic [ 6.103796] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.113204] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.119242] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.123965] mousedev: PS/2 mouse device common for all mice [ 6.128776] rtc_cmos 00:05: RTC can wake from S4 [ 6.137160] rtc_cmos 00:05: registered as rtc0 [ 6.140665] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.141263] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.141302] intel_pstate: CPU model not supported [ 6.154361] hid: raw HID events driver (C) Jiri Kosina [ 6.175226] usbcore: registered new interface driver usbhid [ 6.182767] usbhid: USB HID core driver [ 6.183804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.185200] drop_monitor: Initializing network drop monitor service [ 6.185322] Initializing XFRM netlink socket [ 6.185675] NET: Registered protocol family 10 [ 6.190195] Segment Routing with IPv6 [ 6.190239] NET: Registered protocol family 17 [ 6.191161] mpls_gso: MPLS GSO support [ 6.216200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.229630] RAS: Correctable Errors collector initialized. [ 6.232276] AVX version of gcm_enc/dec engaged. [ 6.234186] AES CTR mode by8 optimization enabled [ 6.372198] sched_clock: Marking stable (6372170641, 0)->(8359803464, -1987632823) [ 6.386328] registered taskstats version 1 [ 6.394746] Loading compiled-in X.509 certificates [ 6.407917] zswap: loaded using pool lzo/zbud [ 6.446416] Key type big_key registered [ 6.459785] Key type encrypted registered [ 6.466594] ima: No TPM chip found, activating TPM-bypass! [ 6.480989] ima: Allocated hash algorithm: sha1 [ 6.485300] ima: No architecture policies found [ 6.492935] evm: Initialising EVM extended attributes: [ 6.499174] evm: security.selinux [ 6.505216] evm: security.ima [ 6.507456] evm: security.capability [ 6.511194] evm: HMAC attrs: 0x1 [ 6.526305] rtc_cmos 00:05: setting system clock to 2026-09-05 00:29:59 UTC (1788568199) [ 6.540314] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 6.550605] debug: unmapping init [mem 0xffffffffb1382000-0xffffffffb1658fff] [ 6.559263] Write protecting the kernel read-only data: 28672k [ 6.570975] debug: unmapping init [mem 0xffffffffafa03000-0xffffffffafbfffff] [ 6.580683] debug: unmapping init [mem 0xffffffffb0314000-0xffffffffb03fffff] [ 6.695558] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.730743] systemd[1]: Detected virtualization kvm. [ 6.737767] systemd[1]: Detected architecture x86-64. [ 6.740652] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.820660] systemd[1]: No hostname configured. [ 6.825371] systemd[1]: Set hostname to . [ 6.832325] random: systemd: uninitialized urandom read (16 bytes read) [ 6.837163] systemd[1]: Initializing machine ID from random generator. [ 6.923468] random: ln: uninitialized urandom read (6 bytes read) [ 7.104533] random: systemd: uninitialized urandom read (16 bytes read) [ 7.111373] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.149990] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.160774] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.809521] device-mapper: uevent: version 1.0.3 [ 8.816699] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.283014] virtio_net virtio0 ens2: renamed from eth0 [ 11.618991] scsi host0: ata_piix [ 11.690566] scsi host1: ata_piix [ 11.692102] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.697497] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.033947] random: crng init done [ 17.035757] random: 7 urandom warning(s) missed due to ratelimiting [ 17.943916] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.376217] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.611870] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.394530] SELinux: Disabled at runtime. [ 23.496118] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.506233] systemd[1]: Detected virtualization kvm. [ 23.508659] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.435467] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.453507] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.468649] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.478119] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.484691] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.501741] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.519404] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 25.102939] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 26.114065] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.681078] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.883621] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.187472] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.316558] EDAC sbridge: Ver: 1.1.2 [ 30.876058] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.884739] NFS: Registering the id_resolver key type [ 31.890104] Key type id_resolver registered [ 31.897734] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg640-client login: [ 88.434927] libcfs: loading out-of-tree module taints kernel. [ 88.664863] Key type ._llcrypt registered [ 88.675609] Key type .llcrypt registered [ 89.514304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.550656] alg: No test for adler32 (adler32-zlib) [ 91.175093] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 92.119757] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 94.009338] Key type lgssc registered [ 96.483738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 225.421418] Lustre: Mounted lustre-client - version 2.17.58_2_gfa0a981 [ 230.205674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.772336] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.073206] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing yml_node [ 250.708152] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 251.361176] Lustre: lustre-OST0000-osc-ffff952c07f20800: disconnect after 24s idle [ 252.734721] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 255.108413] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 256.763441] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 4 20:34:08 EDT 2026 [ 269.414086] Lustre: Unmounted lustre-client [ 303.928557] Key type lgssc unregistered [ 304.167896] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.175108] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 304.209359] LNet: Removed LNI 192.168.206.40@tcp [ 304.988408] Key type .llcrypt unregistered [ 304.999094] Key type ._llcrypt unregistered [ 330.423835] Key type ._llcrypt registered [ 330.425372] Key type .llcrypt registered [ 330.812546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 330.852680] alg: No test for adler32 (adler32-zlib) [ 331.723986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 331.764378] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 331.775901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 335.474338] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 335.838793] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.856953] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.879809] LNet: Removed LNI 192.168.206.40@tcp [ 336.688547] Key type .llcrypt unregistered [ 336.692743] Key type ._llcrypt unregistered [ 342.409279] Key type ._llcrypt registered [ 342.416198] Key type .llcrypt registered [ 343.634502] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:35:35 (1788568535) [ 343.867936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.878521] alg: No test for adler32 (adler32-zlib) [ 344.808464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.911650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.927911] LNet: 9708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.947327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.982581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.999763] LNet: 9715:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 351.807208] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:35:43 (1788568543) [ 351.940931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.995524] LNet: 10290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.035384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.131798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 357.702914] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:35:49 (1788568549) [ 357.905742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.918405] LNet: 10882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.945435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.010763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 364.142481] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:35:55 (1788568555) [ 364.454264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.513966] LNet: 11466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.556400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.633782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 371.610629] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:36:03 (1788568563) [ 371.837733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.861128] LNet: 12051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.888757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.958934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 372.511669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 379.431284] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:36:10 (1788568570) [ 379.543692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.604849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.702485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.278243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.635050] hrtimer: interrupt took 3820525 ns [ 380.927833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 388.111896] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:36:19 (1788568579) [ 388.241623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.296339] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.301602] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 388.369735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.518335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 395.356198] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:36:27 (1788568587) [ 395.520360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.612255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.689847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 395.723990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.219252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 396.268995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.734209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 396.775967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 397.240467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 397.282738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 397.772237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 397.833635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 398.352798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 398.403500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.919521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 398.968282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.461859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 399.506451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.102739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 400.171807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.801611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 400.862450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.441477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 401.503188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 402.101452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 402.132684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.741055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 402.818259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 409.740876] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:36:41 (1788568601) [ 409.903500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.922280] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.926088] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 409.944204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.005187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 410.044869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 416.306528] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:36:47 (1788568607) [ 416.554653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.598325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.674372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.714768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 423.125320] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:36:54 (1788568614) [ 423.244386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.308611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.415877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 423.453409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 430.845534] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:37:02 (1788568622) [ 431.013489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.077706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.158817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.232186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.835082] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:37:09 (1788568629) [ 438.057748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.091587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.163155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 438.205176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 444.200480] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:37:15 (1788568635) [ 444.301264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.337233] LNet: 17427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.348849] LNet: 17427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 444.413359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.494792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 444.521631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 451.012563] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:37:22 (1788568642) [ 451.201204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.244139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.322450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.983039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.254348] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:37:30 (1788568650) [ 459.564748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.610232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.837695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.183701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.825534] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:37:39 (1788568659) [ 467.911154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.989171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.213065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.465707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.440220] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:37:46 (1788568666) [ 474.552807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.594781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.128324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.389257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.572224] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:37:54 (1788568674) [ 482.756518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.873528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.960434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.036530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.081518] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:38:01 (1788568681) [ 490.143921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.217855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.331576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.423200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.428854] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:38:09 (1788568689) [ 497.722210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.783773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.858395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 504.730304] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:38:16 (1788568696) [ 504.937485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.965990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.028319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 511.605401] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:38:23 (1788568703) [ 511.698763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.754666] LNet: 22863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.764387] LNet: 22863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 511.815554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.964922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 512.360468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 518.688324] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:38:30 (1788568710) [ 518.876228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.961756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.051372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 519.090058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 525.519157] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:38:37 (1788568717) [ 525.638324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.712608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.828921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 525.870744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 531.607368] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:38:43 (1788568723) [ 531.743568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.779919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.917482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 531.980262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 538.887024] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:38:50 (1788568730) [ 538.980142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.037210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.145130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 539.191929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 545.647934] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:38:57 (1788568737) [ 545.889744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.993665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.059705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 546.160074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 546.192831] LNetError: 25817:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 546.238697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 546.281380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 552.217778] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:39:04 (1788568744) [ 552.365030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.450585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.592634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zjTQjMIgfJ [ 552.672729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zjTQjMIgfJ [ 552.765888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zjTQjMIgfJ [ 558.431286] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:39:10 (1788568750) [ 558.515239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.566288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.638143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 558.681071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 558.734148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 564.565189] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:39:16 (1788568756) [ 564.790666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.825650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.902616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 564.925066] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 566.508317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 567.831774] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 569.791166] Key type lgssc registered [ 570.754992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.935247] LNetError: 27578:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.40@tcp99 has no available nets [ 576.195521] Key type lgssc unregistered [ 576.550226] LNetError: 28682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.571458] LNet: Removed LNI 192.168.206.40@tcp [ 577.117435] Key type .llcrypt unregistered [ 577.119946] Key type ._llcrypt unregistered [ 582.590811] Key type ._llcrypt registered [ 582.592713] Key type .llcrypt registered [ 583.697929] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:39:35 (1788568775) [ 584.331504] Key type .llcrypt unregistered [ 584.336091] Key type ._llcrypt unregistered [ 590.656717] Key type ._llcrypt registered [ 590.658580] Key type .llcrypt registered [ 590.836029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 590.855600] alg: No test for adler32 (adler32-zlib) [ 591.730796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.806846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.826553] LNet: 29592:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 591.837272] LNet: 29592:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 591.849512] LNet: 29592:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 591.876578] LNet: Message forwarding will be enabled in 2s [ 594.976898] LNet: Message forwarding enabled [ 595.261652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.287093] LNet: Message forwarding will stop in 2s to 6s [ 598.049710] LNet: Message forwarding disabled [ 598.694259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.732106] LNet: 29681:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 598.746476] LNet: 29681:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 598.760278] LNet: 29681:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 598.777828] LNet: Message forwarding will be enabled in 2s [ 598.945671] LNet: 29694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.631532] Key type .llcrypt unregistered [ 599.634269] Key type ._llcrypt unregistered [ 610.130308] Key type ._llcrypt registered [ 610.134374] Key type .llcrypt registered [ 611.490161] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:40:02 (1788568802) [ 611.788769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 611.806708] alg: No test for adler32 (adler32-zlib) [ 612.739843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.808811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 618.588909] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:40:10 (1788568810) [ 618.710243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.744407] LNet: 31087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.765753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.834525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 618.871514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 618.893603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 618.926826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 618.956618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 618.983483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 619.030156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 619.069932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 619.112437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 619.138976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 619.166588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 619.196499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 619.239948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 619.281041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 619.347699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 619.392328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 619.427757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 619.478834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 619.514907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 619.537534] LNetError: 31136:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 625.402328] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:40:17 (1788568817) [ 625.601897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.625215] LNet: 31707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.641269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.686512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.368231] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:40:23 (1788568823) [ 632.884690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg640-server [ 634.688668] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:40:26 (1788568826) [ 634.883363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.912844] LNet: 32636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.949418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.990831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.013837] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 635.810377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[41-45]@tcp [ 635.836475] LNet: 32770:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 635.907392] LNetError: 32641:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.41@tcp is being used as a gateway but routing feature is not turned on [ 635.926517] LNetError: 32641:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.41@tcp has gone from up to down [ 642.707761] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:40:34 (1788568834) [ 644.563286] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg640-server [ 647.053761] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:40:37 (1788568837) [ 647.612530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.685794] LNet: 33706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.712490] LNetError: 33706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.761392] LNet: Removed LNI 192.168.206.40@tcp [ 647.863192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.960921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.012632] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 648.761348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[41-46/2]@tcp [ 648.784438] LNet: 33823:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 648.791849] LNet: 33823:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 648.815336] LNetError: 33710:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.41@tcp is being used as a gateway but routing feature is not turned on [ 648.821260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[41-46/2]@tcp [ 648.830109] LNetError: 33710:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 648.841525] LNetError: 33710:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.41@tcp has gone from up to down [ 648.850144] LNetError: 33710:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 658.098687] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:40:49 (1788568849) [ 658.483483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.526497] LNet: 34417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.535886] LNetError: 34417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.562640] LNet: Removed LNI 192.168.206.40@tcp [ 658.635689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.771650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 658.826568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.870682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.938459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.000388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.044488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.112682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 659.197044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.236044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.284686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.328190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.353318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.393613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 659.454985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.499556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.557044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.657689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.716620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.808407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 659.872734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.908148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.969511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.047854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.073741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.119397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.199667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.266298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.410282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.576958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.654068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.757088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 660.838769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.875168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.933301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 668.415207] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:40:59 (1788568859) [ 670.306216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg640-server [ 671.935727] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:41:03 (1788568863) [ 673.637135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg640-server [ 675.352960] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:41:06 (1788568866) [ 675.516516] LNet: 35815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.521716] LNet: 35815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 676.299945] Key type .llcrypt unregistered [ 676.302231] Key type ._llcrypt unregistered [ 682.655200] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 682.763017] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 682.778549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 682.785371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.440466] Key type ._llcrypt registered [ 685.441991] Key type .llcrypt registered [ 685.731607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.751604] alg: No test for adler32 (adler32-zlib) [ 686.652819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.702454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.719285] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 687.445336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.477175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.550503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.570997] LNet: Removed LNI 10.1.2.3@tcp [ 689.747990] LNet: 36535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.757244] LNetError: 36535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.768917] LNet: Removed LNI 192.168.206.40@tcp [ 690.375576] Key type .llcrypt unregistered [ 690.377842] Key type ._llcrypt unregistered [ 701.623967] Key type ._llcrypt registered [ 701.628635] Key type .llcrypt registered [ 702.929752] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:41:34 (1788568894) [ 703.472405] Key type .llcrypt unregistered [ 703.477318] Key type ._llcrypt unregistered [ 709.007514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.220452] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.227653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.235079] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 711.765580] Key type ._llcrypt registered [ 711.769769] Key type .llcrypt registered [ 711.976993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.005351] alg: No test for adler32 (adler32-zlib) [ 712.953506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.010394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.029841] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 713.754152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.787076] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.099931] LNet: 38024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.105902] LNetError: 38024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 716.111299] LNet: Removed LNI 10.1.2.3@tcp [ 716.124038] LNet: Removed LNI 192.168.206.40@tcp [ 716.703214] Key type .llcrypt unregistered [ 716.704797] Key type ._llcrypt unregistered [ 726.192959] Key type ._llcrypt registered [ 726.197362] Key type .llcrypt registered [ 727.345286] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:41:59 (1788568919) [ 727.933597] Key type .llcrypt unregistered [ 727.936896] Key type ._llcrypt unregistered [ 734.189840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.342990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.357256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.367735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 736.963937] Key type ._llcrypt registered [ 736.966706] Key type .llcrypt registered [ 737.119746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.131817] alg: No test for adler32 (adler32-zlib) [ 738.035812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.098705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 738.132319] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 739.914940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 739.929468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.733826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 741.769591] LNet: Removed LNI 10.1.2.3@tcp [ 741.791595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 741.817067] LNet: Removed LNI 10.1.2.31@tcp [ 741.832217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.854301] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.572670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 742.604886] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 743.607435] LNet: 39936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.611689] LNetError: 39936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 743.614753] LNet: Removed LNI 10.1.2.31@tcp [ 743.622874] LNet: Removed LNI 10.1.2.3@tcp [ 744.302603] Key type .llcrypt unregistered [ 744.305290] Key type ._llcrypt unregistered [ 757.681143] Key type ._llcrypt registered [ 757.687486] Key type .llcrypt registered [ 759.031738] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:42:30 (1788568950) [ 759.501637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.514829] alg: No test for adler32 (adler32-zlib) [ 760.407870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.500674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.530931] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 761.361459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.391196] LNet: 40886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.397218] LNetError: 40886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.409710] LNet: Removed LNI 192.168.206.40@tcp [ 761.438518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.565491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.40@tcp [ 761.608337] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 762.364508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.40@tcp [ 762.395074] LNet: Removed LNI 192.168.206.40@tcp [ 762.484800] LNet: 41021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.128451] Key type .llcrypt unregistered [ 763.135139] Key type ._llcrypt unregistered [ 775.398505] Key type ._llcrypt registered [ 775.401726] Key type .llcrypt registered [ 776.608404] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:42:48 (1788568968) [ 778.426390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg640-server [ 779.983780] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:42:51 (1788568971) [ 780.559418] Key type .llcrypt unregistered [ 780.566436] Key type ._llcrypt unregistered [ 786.616491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 786.838172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 786.846628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 786.857042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 789.466431] Key type ._llcrypt registered [ 789.470942] Key type .llcrypt registered [ 789.701477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.717126] alg: No test for adler32 (adler32-zlib) [ 790.644666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.698939] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 791.545472] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 793.909873] LNet: 42848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.921586] LNetError: 42848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.934073] LNet: Removed LNI 192.168.206.40@tcp1 [ 793.956227] LNet: Removed LNI 10.1.2.3@tcp2 [ 794.594374] Key type .llcrypt unregistered [ 794.596261] Key type ._llcrypt unregistered [ 804.531374] Key type ._llcrypt registered [ 804.533442] Key type .llcrypt registered [ 805.877986] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:43:17 (1788568997) [ 806.447339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.464787] alg: No test for adler32 (adler32-zlib) [ 807.412805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.489086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.509758] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 808.297054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.323739] LNet: 43814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.331673] LNetError: 43814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.352795] LNet: Removed LNI 192.168.206.40@tcp [ 808.389903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.455511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 808.497173] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 815.216662] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:43:26 (1788569006) [ 815.372405] LNet: 44402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.383874] LNetError: 44402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.407258] LNet: Removed LNI 192.168.206.40@tcp [ 816.042957] Key type .llcrypt unregistered [ 816.052824] Key type ._llcrypt unregistered [ 822.727991] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.873172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.876930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.880332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 825.455730] Key type ._llcrypt registered [ 825.458168] Key type .llcrypt registered [ 825.676195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.701913] alg: No test for adler32 (adler32-zlib) [ 826.701702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.782800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.817600] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 827.647728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 827.672182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.812769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.825846] LNet: 45144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.830874] LNetError: 45144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.835521] LNet: Removed LNI 10.1.2.3@tcp [ 829.848861] LNet: Removed LNI 192.168.206.40@tcp [ 829.877312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.995171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.031916] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 830.041991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.647799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.670861] LNet: 45171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.678684] LNetError: 45171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.686169] LNetError: 45171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 830.693462] LNet: Removed LNI 10.1.2.3@tcp [ 830.719316] LNet: Removed LNI 192.168.206.40@tcp [ 830.766476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.840389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.865549] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 830.871822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.297245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 831.321235] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 832.143590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.162777] LNet: 45334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.168373] LNetError: 45334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.174659] LNetError: 45334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 832.180614] LNet: Removed LNI 10.1.2.3@tcp [ 832.183606] LNet: Removed LNI 192.168.206.40@tcp [ 832.195225] LNet: Removed LNI 192.168.206.40@tcp2 [ 832.216504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.285877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 832.304178] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 832.307946] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 832.313593] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 838.831782] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:43:50 (1788569030) [ 838.907684] LNet: 45922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.919112] LNetError: 45922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 838.926448] LNetError: 45922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 838.938108] LNet: Removed LNI 10.1.2.3@tcp [ 838.940230] LNet: Removed LNI 192.168.206.40@tcp [ 838.959767] LNet: Removed LNI 192.168.206.40@tcp2 [ 839.519768] Key type .llcrypt unregistered [ 839.523088] Key type ._llcrypt unregistered [ 845.371790] Key type ._llcrypt registered [ 845.377681] Key type .llcrypt registered [ 845.618523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 845.637587] alg: No test for adler32 (adler32-zlib) [ 846.545710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.621262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.656546] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 847.507338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.536206] LNet: 46322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.546115] LNetError: 46322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.558491] LNet: Removed LNI 192.168.206.40@tcp [ 847.592908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.690753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.713042] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 848.066739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.082892] LNet: 46352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.088883] LNetError: 46352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.096623] LNet: Removed LNI 192.168.206.40@tcp [ 848.133590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.227439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.250642] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 848.784061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.818099] LNet: Removed LNI 192.168.206.40@tcp [ 848.836451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.919558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.945772] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 849.576259] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.828237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.834763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.844850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.221251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.240657] LNet: 46462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.243977] LNet: 46462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 852.248692] LNetError: 46462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.253846] LNetError: 46462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 852.278556] LNet: Removed LNI 192.168.206.40@tcp [ 852.296812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.368828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 852.398876] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 858.805690] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:44:10 (1788569050) [ 858.925140] LNet: 47049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.932835] LNetError: 47049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.946772] LNet: Removed LNI 192.168.206.40@tcp [ 859.561792] Key type .llcrypt unregistered [ 859.563571] Key type ._llcrypt unregistered [ 865.728807] Key type ._llcrypt registered [ 865.731678] Key type .llcrypt registered [ 865.876830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 865.892052] alg: No test for adler32 (adler32-zlib) [ 866.870108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.955587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.987113] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 868.007738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.027696] LNet: 47453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.033766] LNetError: 47453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.056686] LNet: Removed LNI 192.168.206.40@tcp [ 868.078140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.137517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 868.165968] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 875.105289] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:44:26 (1788569066) [ 875.203934] LNet: 48037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.210321] LNetError: 48037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.229842] LNet: Removed LNI 192.168.206.40@tcp [ 876.024325] Key type .llcrypt unregistered [ 876.027239] Key type ._llcrypt unregistered [ 883.198039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.361282] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 883.364891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 883.371658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.057275] Key type ._llcrypt registered [ 886.064888] Key type .llcrypt registered [ 886.305582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.318869] alg: No test for adler32 (adler32-zlib) [ 887.225123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.291212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.311167] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 888.552428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.584225] LNet: 48488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.595499] LNetError: 48488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.606154] LNet: Removed LNI 192.168.206.40@tcp [ 888.621390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.730836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 888.756939] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 895.863904] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:44:47 (1788569087) [ 896.042834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.091921] LNet: 49085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.112331] LNetError: 49085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.144605] LNet: Removed LNI 192.168.206.40@tcp [ 896.166490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.268872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 901.989482] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:44:53 (1788569093) [ 902.148818] LNet: 49667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.876546] Key type .llcrypt unregistered [ 902.878303] Key type ._llcrypt unregistered [ 909.339525] Key type ._llcrypt registered [ 909.342416] Key type .llcrypt registered [ 909.519477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 909.534823] alg: No test for adler32 (adler32-zlib) [ 910.457727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.514557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.560131] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 911.398417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.418632] LNet: 50066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.424446] LNetError: 50066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.442192] LNet: Removed LNI 192.168.206.40@tcp [ 911.477129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.536753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 911.569942] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 917.537125] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:45:09 (1788569109) [ 917.659456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.679782] LNet: 50654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.690537] LNetError: 50654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.722563] LNet: Removed LNI 192.168.206.40@tcp [ 917.767422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.834799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 923.886935] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:45:15 (1788569115) [ 925.727949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 927.451475] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:45:19 (1788569119) [ 929.487267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 931.312843] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:45:22 (1788569122) [ 932.991060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 934.980031] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:45:26 (1788569126) [ 936.467361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 938.241261] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:45:29 (1788569129) [ 939.992976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 941.689879] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:45:33 (1788569133) [ 941.799168] LNet: 52706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.390566] Key type .llcrypt unregistered [ 942.393777] Key type ._llcrypt unregistered [ 948.936483] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.077273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.095182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.111128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.814146] Key type ._llcrypt registered [ 951.820016] Key type .llcrypt registered [ 952.011275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.021088] alg: No test for adler32 (adler32-zlib) [ 953.001922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.125973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.168721] LNet: Added LNI 192.168.206.40@tcp [8/512/0/180] [ 953.175891] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 953.180343] LNet: 53029:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.263813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.298489] LNet: 53048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.307551] LNetError: 53048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.318724] LNet: Removed LNI 10.1.2.3@tcp [ 953.320963] LNetError: 53048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.344076] LNet: Removed LNI 192.168.206.40@tcp [ 953.377731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.434206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.451513] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 953.514673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.560803] LNet: 53072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.566981] LNetError: 53072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.583200] LNet: Removed LNI 192.168.206.40@tcp [ 953.601937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.666486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.687258] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 953.691653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.693461] LNet: 53080:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.779666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.798229] LNet: 53098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.803923] LNetError: 53098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.808946] LNet: Removed LNI 10.1.2.3@tcp [ 953.811683] LNetError: 53098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.824480] LNet: Removed LNI 192.168.206.40@tcp [ 953.846698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.936924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.976912] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 953.980535] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.982395] LNet: 53106:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 954.238706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.266943] LNet: 53132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.283453] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.292514] LNet: Removed LNI 10.1.2.3@tcp [ 954.296438] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.309952] LNet: Removed LNI 192.168.206.40@tcp [ 954.324318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.385488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.434515] LNet: Added LNI 192.168.206.40@tcp [8/256/0/182] [ 954.443111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.565541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.588423] LNet: 53161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.598135] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.614160] LNet: Removed LNI 10.1.2.3@tcp [ 954.619345] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.653736] LNet: Removed LNI 192.168.206.40@tcp [ 954.686241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.789944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.826767] LNet: Added LNI 192.168.206.40@tcp [8/256/0/182] [ 954.835423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.848145] LNet: 53169:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 955.013475] LNet: Removed LNI 10.1.2.3@tcp [ 955.029976] LNet: Removed LNI 192.168.206.40@tcp [ 955.594796] Key type .llcrypt unregistered [ 955.599221] Key type ._llcrypt unregistered [ 966.801145] Key type ._llcrypt registered [ 966.804904] Key type .llcrypt registered [ 968.070070] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 20:45:59 (1788569159) [ 969.524995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 970.930913] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:46:02 (1788569162) [ 971.482829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.494872] alg: No test for adler32 (adler32-zlib) [ 972.408919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.505644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 972.529297] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 973.252024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.40 [ 973.273552] LNet: 54429:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 973.282068] LNet: Removed LNI 192.168.206.40@tcp [ 979.239959] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:46:10 (1788569170) [ 979.437600] LNet: 54995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.450905] LNetError: 54995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.485959] LNet: Removed LNI 192.168.206.40@tcp [ 980.033362] Key type .llcrypt unregistered [ 980.041250] Key type ._llcrypt unregistered [ 985.786091] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.028981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 987.582829] Key type ._llcrypt registered [ 987.585907] Key type .llcrypt registered [ 987.756471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.773297] alg: No test for adler32 (adler32-zlib) [ 989.236822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.866813] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:46:26 (1788569186) [ 995.070231] LNet: 55900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.080082] LNetError: 55900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 995.090738] LNet: Removed LNI 10.1.2.3@tcp [ 995.730417] Key type .llcrypt unregistered [ 995.732380] Key type ._llcrypt unregistered [ 1001.508620] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1001.759643] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1001.765058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1001.767616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1004.539234] Key type ._llcrypt registered [ 1004.547028] Key type .llcrypt registered [ 1004.762854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.783760] alg: No test for adler32 (adler32-zlib) [ 1005.772797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1011.747742] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:46:43 (1788569203) [ 1013.028377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1014.555716] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:46:46 (1788569206) [ 1014.774728] LNet: 57082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.786394] LNetError: 57082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.806901] LNet: Removed LNI 10.1.2.3@tcp [ 1015.277771] Key type .llcrypt unregistered [ 1015.281213] Key type ._llcrypt unregistered [ 1020.552399] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.794796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.801037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.809372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1023.479897] Key type ._llcrypt registered [ 1023.482136] Key type .llcrypt registered [ 1023.681343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.700820] alg: No test for adler32 (adler32-zlib) [ 1024.599802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.664417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1032.821643] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:47:04 (1788569224) [ 1033.383597] LNet: 58309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.390143] LNetError: 58309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1033.413376] LNet: Removed LNI 10.1.2.3@tcp [ 1034.098850] Key type .llcrypt unregistered [ 1034.105516] Key type ._llcrypt unregistered [ 1040.123921] Key type ._llcrypt registered [ 1040.128237] Key type .llcrypt registered [ 1040.344487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.358021] alg: No test for adler32 (adler32-zlib) [ 1041.285537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1041.315516] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1048.818161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1049.606668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_interrupt -r 1 [ 1049.646969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_interrupt -r 1 [ 1049.689966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1049.710332] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1050.005608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.051159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.097263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.145393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.200676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1050.252760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.288996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.330949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.376415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.416580] LNet: 58746:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.441647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.480217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.519178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.527341] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1050.550205] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1050.571547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.605564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.631955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.660461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1051.124800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_dropped -r 1 [ 1051.153304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_dropped -r 1 [ 1051.203708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1051.473948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.512268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.542298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.575224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.598391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1051.637966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.665773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.710914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.758668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.790895] LNet: 58858:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.812090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.872443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.910393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.935649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.967944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.995521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.025529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.528819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_aborted -r 1 [ 1052.563680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_aborted -r 1 [ 1052.603069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1052.659767] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1052.674732] LNet: Skipped 1 previous similar message [ 1052.972136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.007714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.040625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.081166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.120136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1053.161608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.199090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.242853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.284309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.309614] LNet: 58969:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.328637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.389665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.443256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.476886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.515617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.552951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.603362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1054.241677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_no_route -r 1 [ 1054.283458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_no_route -r 1 [ 1054.351839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1054.583648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.610401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.634323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.658807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.685545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1054.723757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.747732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.785888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.826729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.855649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.908598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.939572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.970473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.001649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.034318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.056605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1055.635336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_timeout -r 1 [ 1055.666656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_timeout -r 1 [ 1055.701555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1055.720327] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1055.726962] LNet: Skipped 1 previous similar message [ 1055.916243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.944474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1056.009820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1056.052326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.117447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1056.158934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.188532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.229459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.276727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.300241] LNet: 59192:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.305692] LNet: 59192:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1056.349220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.392380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.459698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.518232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.565522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.607642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.664577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1057.247661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_error -r 1 [ 1057.283085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_error -r 1 [ 1057.339475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1057.650199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.678583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.710479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.736749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.761782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1057.796920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.820092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.848438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.895152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.938701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.990593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.030267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.084467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.113790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.152986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.182646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1058.466691] LNet: 59344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.472492] LNetError: 59344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.489463] LNet: Removed LNI 192.168.206.40@tcp [ 1058.955727] Key type .llcrypt unregistered [ 1058.958203] Key type ._llcrypt unregistered [ 1074.682647] Key type ._llcrypt registered [ 1074.689537] Key type .llcrypt registered [ 1075.860994] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:47:47 (1788569267) [ 1076.557356] Key type .llcrypt unregistered [ 1076.563969] Key type ._llcrypt unregistered [ 1082.409991] Key type ._llcrypt registered [ 1082.412423] Key type .llcrypt registered [ 1082.644304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.673828] alg: No test for adler32 (adler32-zlib) [ 1083.606618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.653800] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1091.068369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1101.571464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.613224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1101.647721] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1103.012748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_interrupt -r 1 [ 1103.068532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_interrupt -r 1 [ 1103.158884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_interrupt -r 1 [ 1103.237865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_interrupt -r 1 [ 1103.306824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_interrupt -r 1 [ 1103.366938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_interrupt -r 1 [ 1103.406409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_interrupt -r 1 [ 1103.461388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_interrupt -r 1 [ 1103.496422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1103.529451] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1103.541315] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1103.800913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.827846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.854708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.877878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.901765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1103.943719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.977756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.020823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.059641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.083809] LNet: 60810:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.112654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.141877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.179591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.215480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.248389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.284724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.324856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.811915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_dropped -r 1 [ 1104.845202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_dropped -r 1 [ 1104.886886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_dropped -r 1 [ 1104.932410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_dropped -r 1 [ 1104.973934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_dropped -r 1 [ 1105.012113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_dropped -r 1 [ 1105.050206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_dropped -r 1 [ 1105.076883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_dropped -r 1 [ 1105.110499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1105.132838] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1105.142786] LNet: Skipped 2 previous similar messages [ 1105.396492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.423936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.448649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.472815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.496153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1105.521890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.563508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.616934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.651320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.676471] LNet: 60933:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.701726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.748375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.793450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.821873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.862326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.901809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.940677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.466356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_aborted -r 1 [ 1106.495672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_aborted -r 1 [ 1106.538238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_aborted -r 1 [ 1106.581614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_aborted -r 1 [ 1106.623488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_aborted -r 1 [ 1106.654845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_aborted -r 1 [ 1106.688442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_aborted -r 1 [ 1106.726222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_aborted -r 1 [ 1106.771534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1106.800130] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1106.804957] LNet: Skipped 2 previous similar messages [ 1107.088678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.116407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.139753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.161848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.216978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1107.247549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.271714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.303198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.335776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.354072] LNet: 61056:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.372478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.416976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.456671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.489569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.533225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.586052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.616956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.217483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_no_route -r 1 [ 1108.281777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_no_route -r 1 [ 1108.319217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_no_route -r 1 [ 1108.356674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_no_route -r 1 [ 1108.388684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_no_route -r 1 [ 1108.427528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_no_route -r 1 [ 1108.468079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_no_route -r 1 [ 1108.511347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_no_route -r 1 [ 1108.545979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1108.789803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.833364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.865711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.896498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.935991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1108.980484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.018597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.060660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.086944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.117892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.147800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.185433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.218467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.244617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.284400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.308739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.944884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_timeout -r 1 [ 1109.990809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_timeout -r 1 [ 1110.025941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_timeout -r 1 [ 1110.060202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_timeout -r 1 [ 1110.093841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_timeout -r 1 [ 1110.124706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_timeout -r 1 [ 1110.155848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_timeout -r 1 [ 1110.187383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_timeout -r 1 [ 1110.223958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1110.243765] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1110.248594] LNet: Skipped 5 previous similar messages [ 1110.467912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.490932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.514995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.578408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.650387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.699975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.774597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.825263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.874283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.903229] LNet: 61303:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.913316] LNet: 61303:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1110.944725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.986748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.026808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.055284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.089503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.119314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.148677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.732902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_error -r 1 [ 1111.762663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_error -r 1 [ 1111.794345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_error -r 1 [ 1111.822626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_error -r 1 [ 1111.853510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_error -r 1 [ 1111.883930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_error -r 1 [ 1111.950280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_error -r 1 [ 1112.007759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_error -r 1 [ 1112.054823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1112.380952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.438095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.475035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.508630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.531832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1112.559673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.592052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.632645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.660150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.695779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.728800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.754645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.782695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.818874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.855915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.878688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.594607] LNet: 61503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.599850] LNetError: 61503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.604749] LNet: Removed LNI 192.168.206.40@tcp [ 1118.620624] LNet: Removed LNI 192.168.206.40@tcp1 [ 1119.076693] Key type .llcrypt unregistered [ 1119.080751] Key type ._llcrypt unregistered [ 1134.178220] Key type ._llcrypt registered [ 1134.181089] Key type .llcrypt registered [ 1135.576255] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:48:47 (1788569327) [ 1136.398255] Key type .llcrypt unregistered [ 1136.401534] Key type ._llcrypt unregistered [ 1142.204697] Key type ._llcrypt registered [ 1142.212221] Key type .llcrypt registered [ 1142.417698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.445298] alg: No test for adler32 (adler32-zlib) [ 1143.387900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.418493] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1151.638442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1152.352214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_dropped -r 1 [ 1152.389858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_dropped -r 1 [ 1152.442060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1152.484817] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1152.797518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.834975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.864426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.888183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.921249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1152.978370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.025231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.054686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.103773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.131357] LNet: 62750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.158638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.210723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.250634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.282351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.324494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.367084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.424888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.979383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_error -r 1 [ 1154.014103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_error -r 1 [ 1154.058343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1154.075740] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1154.257702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.291368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.316975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.343773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.371485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1154.392643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.421497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.452151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.482819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.502886] LNet: 62861:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.526585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.556839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.578990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.602165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.643378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.675453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.709507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.151330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_timeout -r 1 [ 1155.185927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_timeout -r 1 [ 1155.220520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1155.238818] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1155.429903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.470635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.501992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.532503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.564088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1155.586766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.614217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.646263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.688731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.705266] LNet: 62972:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.722693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.753833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.786573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.813918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.867702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.906844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.937422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.199634] LNet: 63013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.213527] LNetError: 63013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.226687] LNet: Removed LNI 192.168.206.40@tcp [ 1156.663414] Key type .llcrypt unregistered [ 1156.685359] Key type ._llcrypt unregistered [ 1171.496643] Key type ._llcrypt registered [ 1171.504044] Key type .llcrypt registered [ 1172.981889] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:49:24 (1788569364) [ 1173.834323] Key type .llcrypt unregistered [ 1173.836036] Key type ._llcrypt unregistered [ 1180.173185] Key type ._llcrypt registered [ 1180.174966] Key type .llcrypt registered [ 1180.437933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.453545] alg: No test for adler32 (adler32-zlib) [ 1181.338657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1181.364615] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1188.306956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1195.915269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.967154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1195.981650] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1197.188902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_dropped -r 1 [ 1197.224236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e remote_dropped -r 1 [ 1197.253772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_dropped -r 1 [ 1197.287879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e remote_dropped -r 1 [ 1197.341166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e remote_dropped -r 1 [ 1197.389219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e remote_dropped -r 1 [ 1197.434817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e remote_dropped -r 1 [ 1197.485394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e remote_dropped -r 1 [ 1197.525803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1197.539753] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1197.544595] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.140@tcp [ 1197.737180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.786286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.812091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.854584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.894647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1197.920808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.954266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.022621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.071403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.107507] LNet: 64476:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.121619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.158487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.194341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.251644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.271565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.294881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.319685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.369343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.908147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_error -r 1 [ 1198.950647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e remote_error -r 1 [ 1198.999796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_error -r 1 [ 1199.039875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e remote_error -r 1 [ 1199.071369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e remote_error -r 1 [ 1199.104907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e remote_error -r 1 [ 1199.145573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e remote_error -r 1 [ 1199.182902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e remote_error -r 1 [ 1199.235267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1199.267474] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1199.275456] LNet: Skipped 2 previous similar messages [ 1199.550923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.577723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.600814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.626368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.651646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1199.689867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.740412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.792545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.834680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.869853] LNet: 64602:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.897872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.937341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.997487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.034547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.069575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.114398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.165951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.210670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.759777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_timeout -r 1 [ 1200.793619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e remote_timeout -r 1 [ 1200.824475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_timeout -r 1 [ 1200.851196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e remote_timeout -r 1 [ 1200.881654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e remote_timeout -r 1 [ 1200.907894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e remote_timeout -r 1 [ 1200.936620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e remote_timeout -r 1 [ 1200.975545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e remote_timeout -r 1 [ 1201.027276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.140@tcp [ 1201.058839] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1201.379647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.413454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.447274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.482819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.548520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1201.600439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.643708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.668890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.718954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.752834] LNet: 64727:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.779222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.825951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.864690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.895961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.935282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.965721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.013542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.040929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.268477] LNet: 64805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.272545] LNetError: 64805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.276826] LNet: Removed LNI 192.168.206.40@tcp [ 1207.286548] LNet: Removed LNI 192.168.206.40@tcp1 [ 1207.673797] Key type .llcrypt unregistered [ 1207.678471] Key type ._llcrypt unregistered [ 1221.947506] Key type ._llcrypt registered [ 1221.950114] Key type .llcrypt registered [ 1223.061410] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:50:14 (1788569414) [ 1224.804808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1226.939764] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:50:18 (1788569418) [ 1228.010462] Key type .llcrypt unregistered [ 1228.018748] Key type ._llcrypt unregistered [ 1234.100166] Key type ._llcrypt registered [ 1234.104855] Key type .llcrypt registered [ 1234.340136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.361399] alg: No test for adler32 (adler32-zlib) [ 1235.260097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1235.278788] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1243.764441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1244.467524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e network_timeout -r 1 [ 1244.497470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e network_timeout -r 1 [ 1244.531475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1244.552556] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1244.799919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.867921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.928288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.976074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.054309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1245.104267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.192362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.249972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.304720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.328569] LNet: 66345:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.356512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.400476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.467135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.512984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.539495] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1245.549165] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1245.566554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.610572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.836145] LNet: 66384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.840703] LNetError: 66384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.855840] LNet: Removed LNI 192.168.206.40@tcp [ 1246.159555] Key type .llcrypt unregistered [ 1246.162413] Key type ._llcrypt unregistered [ 1261.732070] Key type ._llcrypt registered [ 1261.733486] Key type .llcrypt registered [ 1261.868389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1261.893458] alg: No test for adler32 (adler32-zlib) [ 1262.849864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1262.873072] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1269.683376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1278.675476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.702527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1278.721128] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1279.946313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e network_timeout -r 1 [ 1279.977826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e network_timeout -r 1 [ 1280.011939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e network_timeout -r 1 [ 1280.053297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e network_timeout -r 1 [ 1280.095194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e network_timeout -r 1 [ 1280.130240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e network_timeout -r 1 [ 1280.165405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e network_timeout -r 1 [ 1280.202331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e network_timeout -r 1 [ 1280.236445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1280.254817] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1280.288713] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1280.290813] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.140@tcp [ 1280.441498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.480205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.513203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.561480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.582715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.614739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.658977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.724517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.765211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.780753] LNet: 67275:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.795842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.825545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.849479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.892573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.930797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.974641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.539847] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1285.873882] LNet: 67348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.885031] LNetError: 67348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.890755] LNet: Removed LNI 192.168.206.40@tcp [ 1285.906564] LNet: Removed LNI 192.168.206.40@tcp1 [ 1286.337567] Key type .llcrypt unregistered [ 1286.340228] Key type ._llcrypt unregistered [ 1300.599369] Key type ._llcrypt registered [ 1300.603559] Key type .llcrypt registered [ 1301.994877] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:51:33 (1788569493) [ 1302.288801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1302.299601] alg: No test for adler32 (adler32-zlib) [ 1303.194202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.257799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1303.303453] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1303.923447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1303.938776] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1304.544224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 1304.614275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1304.665648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1304.691851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1304.726752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1304.764491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1304.796250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.40@tcp [ 1304.820776] LNetError: 68444:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.40@tcp to recovery [ 1305.311314] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1305.314061] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1307.359674] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1308.258341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1311.455502] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1311.745533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1311.773901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1311.816421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.829809] LNet: 68551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.834976] LNetError: 68551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.840414] LNet: Removed LNI 192.168.206.40@tcp [ 1311.858880] LNet: Removed LNI 192.168.206.40@tcp1 [ 1311.880223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.952934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1311.972654] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1312.750702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.770442] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1313.630427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 1313.690784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.753887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1313.807134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1313.876836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1313.903533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1313.934066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1313.973412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.40@tcp [ 1314.004126] LNetError: 68824:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.40@tcp to recovery [ 1314.975412] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1317.559606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1321.119336] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1321.125140] LNet: Skipped 1 previous similar message [ 1325.644616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1325.674881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1325.698566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.335488] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:52:04 (1788569524) [ 1332.480142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.504777] LNet: 69539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.522708] LNetError: 69539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.531227] LNetError: 69539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1332.540554] LNet: Removed LNI 192.168.206.40@tcp [ 1332.572459] LNet: Removed LNI 192.168.206.40@tcp1 [ 1332.594330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.649695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.669749] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1333.549257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.576956] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1334.324179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 1334.384922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1334.412741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1334.443626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1334.469142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1334.494875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1334.528256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.40@tcp --health 0 [ 1334.608557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1334.687289] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.40@tcp [ 1335.720465] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1335.736787] LNet: Skipped 1 previous similar message [ 1350.031689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.148174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1350.193423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.248172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.40@tcp --health 500 [ 1350.359424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.170875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.218965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.234171] LNet: 70033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.238834] LNetError: 70033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.245301] LNetError: 70033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.250137] LNet: Removed LNI 192.168.206.40@tcp [ 1357.260576] LNet: Removed LNI 192.168.206.40@tcp1 [ 1357.291468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.354024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.381270] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1358.148185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.180723] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1358.999932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 1359.062241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.134434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1359.171441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1359.219177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1359.268422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1359.328407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1359.401809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.40@tcp --health 0 [ 1360.416835] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1360.427486] LNet: Skipped 3 previous similar messages [ 1360.718612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.031717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.075719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1371.118806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.835141] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:52:49 (1788569569) [ 1378.276486] LNet: 71079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.286484] LNetError: 71079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.292330] LNetError: 71079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1378.308062] LNet: Removed LNI 192.168.206.40@tcp [ 1378.324378] LNet: Removed LNI 192.168.206.40@tcp1 [ 1378.887486] Key type .llcrypt unregistered [ 1378.891818] Key type ._llcrypt unregistered [ 1385.179884] Key type ._llcrypt registered [ 1385.184761] Key type .llcrypt registered [ 1385.452175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.493521] alg: No test for adler32 (adler32-zlib) [ 1386.426146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1386.456872] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1394.491395] Lustre: DEBUG MARKER: Initial discovery [ 1394.549392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.140@tcp [ 1396.680468] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1396.724498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1396.761932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.140@tcp [ 1396.791600] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1396.811255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1396.846434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.140@tcp -r 1 -m GET -l 3 [ 1396.902622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.140@tcp -r 1 -m GET -e local_error [ 1396.937580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.140@tcp -r 1 -m PUT -l 6 [ 1397.731140] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1398.092650] Lustre: DEBUG MARKER: Force oleg640-client.virtnet to discover 192.168.206.140@tcp (in background) [ 1398.274680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.140@tcp [ 1399.625633] Lustre: DEBUG MARKER: Force oleg640-server to discover 192.168.206.40@tcp [ 1401.375323] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1401.629807] Lustre: DEBUG MARKER: Wait for 71839 [ 1403.042315] Lustre: DEBUG MARKER: Finished wait on 71839 [ 1403.091268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1403.120435] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1403.130941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1403.358628] LNet: 72286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.367046] LNetError: 72286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.384344] LNet: Removed LNI 192.168.206.40@tcp [ 1403.741151] Key type .llcrypt unregistered [ 1403.746741] Key type ._llcrypt unregistered [ 1418.774427] Key type ._llcrypt registered [ 1418.776617] Key type .llcrypt registered [ 1420.036871] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:53:31 (1788569611) [ 1420.437341] Key type .llcrypt unregistered [ 1420.441863] Key type ._llcrypt unregistered [ 1426.656810] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1426.892874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1426.899037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1426.908488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1429.447971] Key type ._llcrypt registered [ 1429.450214] Key type .llcrypt registered [ 1429.603707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1429.619966] alg: No test for adler32 (adler32-zlib) [ 1430.538568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.587027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.623979] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1431.348605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1431.370225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1438.540869] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:53:50 (1788569630) [ 1438.652217] LNet: 74348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.664636] LNetError: 74348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1438.670078] LNet: Removed LNI 10.1.2.3@tcp [ 1438.693727] LNet: Removed LNI 192.168.206.40@tcp [ 1439.318434] Key type .llcrypt unregistered [ 1439.321431] Key type ._llcrypt unregistered [ 1445.723399] Key type ._llcrypt registered [ 1445.730311] Key type .llcrypt registered [ 1445.925886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1445.941621] alg: No test for adler32 (adler32-zlib) [ 1446.866620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.921891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.951889] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1447.868853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1447.900603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1448.979352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.40@tcp [ 1450.015265] LNetError: Unexpected error -22 connecting to 192.168.206.40@tcp at host 192.168.206.40:988 [ 1454.082521] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:54:05 (1788569645) [ 1454.205320] LNet: 75479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.211970] LNetError: 75479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1454.219498] LNet: Removed LNI 10.1.2.3@tcp [ 1454.230847] LNet: Removed LNI 192.168.206.40@tcp [ 1454.807546] Key type .llcrypt unregistered [ 1454.811089] Key type ._llcrypt unregistered [ 1460.171118] Key type ._llcrypt registered [ 1460.172980] Key type .llcrypt registered [ 1460.363114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1460.379434] alg: No test for adler32 (adler32-zlib) [ 1461.318737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.390326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1461.419509] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1462.146085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1462.165928] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 1463.067835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.40@tcp1 --nid 192.168.206.40@tcp2 [ 1467.315144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp1 192.168.206.40@tcp2 [ 1467.332451] LNetError: 76318:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.40@tcp1 to 192.168.206.40@tcp2 - Route table may be misconfigured [ 1467.354275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp2 192.168.206.40@tcp1 [ 1473.037729] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:54:25 (1788569665) [ 1473.164317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.193159] LNet: 76889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.204721] LNetError: 76889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.213807] LNet: Removed LNI 192.168.206.40@tcp1 [ 1474.279708] LNet: Removed LNI 192.168.206.40@tcp2 [ 1474.341648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.428049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.464759] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1475.088860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1475.111931] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1475.978634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 1476.005213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.40@tcp [ 1476.031171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp1 192.168.206.40@tcp1 [ 1476.083917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.40@tcp -d 192.168.206.40@tcp -e network_timeout [ 1476.116824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -e network_timeout [ 1476.154780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -e network_timeout [ 1476.191293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -e network_timeout [ 1476.228122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1476.251979] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1476.271394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1476.368923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1476.440162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1476.447166] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1481.770928] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:54:33 (1788569673) [ 1481.914780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.938631] LNet: 77750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.957961] LNetError: 77750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.969979] LNetError: 77750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.985364] LNet: Removed LNI 192.168.206.40@tcp [ 1483.047718] LNet: Removed LNI 192.168.206.40@tcp1 [ 1483.130375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.218485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1483.507694] LNet: 77789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.877172] Key type .llcrypt unregistered [ 1483.885160] Key type ._llcrypt unregistered [ 1495.303131] Key type ._llcrypt registered [ 1495.304594] Key type .llcrypt registered [ 1496.710977] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:54:48 (1788569688) [ 1497.211715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.225293] alg: No test for adler32 (adler32-zlib) [ 1498.173247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.572148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1501.171100] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:54:52 (1788569692) [ 1501.293944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.323835] LNet: 78882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.366737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.424341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.449554] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1502.246495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.278670] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1503.021121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1503.071076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp1 [ 1503.117623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp1 [ 1508.552503] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:55:00 (1788569700) [ 1509.892442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg640-server [ 1511.215721] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:55:03 (1788569703) [ 1512.770081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg640-server [ 1514.135893] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:55:06 (1788569706) [ 1515.753555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg640-server [ 1517.253130] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:55:08 (1788569708) [ 1518.904766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg640-server [ 1520.349605] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:55:12 (1788569712) [ 1522.000706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg640-server [ 1523.407874] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:55:15 (1788569715) [ 1524.949290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg640-server [ 1526.266097] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:55:18 (1788569718) [ 1527.757288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg640-server [ 1529.249833] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:55:20 (1788569720) [ 1530.772986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg640-server [ 1532.394181] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:55:24 (1788569724) [ 1534.438559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg640-server [ 1536.171827] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:55:27 (1788569727) [ 1536.228892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.243229] LNet: 82757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.250895] LNetError: 82757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.313442] LNet: Removed LNI 192.168.206.40@tcp [ 1537.315485] LNetError: 82757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.333990] LNet: Removed LNI 192.168.206.40@tcp1 [ 1537.381483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.477987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.507292] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1538.117932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1538.209532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1538.380385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.412754] LNet: 82909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.419309] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.495075] LNet: Removed LNI 192.168.206.40@tcp [ 1539.546618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.607201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1539.639871] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1540.266551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1540.472516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.499945] LNet: 83054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.508329] LNetError: 83054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.539128] LNet: Removed LNI 192.168.206.40@tcp [ 1540.566330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.665310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.716960] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1541.445986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1541.558634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1541.773959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.826777] LNet: Removed LNI 192.168.206.40@tcp [ 1542.870994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.000458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1543.026085] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1543.946906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1544.147680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.192330] LNet: Removed LNI 192.168.206.40@tcp [ 1545.239383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.337797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.380321] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1546.145846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1546.232533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1547.588395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.620862] LNet: 83503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.639350] LNet: 83503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1547.663482] LNetError: 83503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.672669] LNetError: 83503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1547.704549] LNet: Removed LNI 192.168.206.40@tcp [ 1547.739161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.806372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1547.854063] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1548.660448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1551.161991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.229786] LNet: Removed LNI 192.168.206.40@tcp [ 1552.251287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.309965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.332149] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1553.045956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1553.171598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1553.412430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.471565] LNet: Removed LNI 192.168.206.40@tcp [ 1554.518432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.560358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1554.590290] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1555.184423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1555.384313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.461230] LNet: Removed LNI 192.168.206.40@tcp [ 1556.493295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.562609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1556.590769] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1557.278271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1557.353839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1557.650220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.675476] LNet: 84107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.681477] LNet: 84107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1557.690944] LNetError: 84107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.700703] LNetError: 84107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1558.764252] LNet: Removed LNI 192.168.206.40@tcp [ 1558.793073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.884560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1558.922704] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1559.681784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1559.894529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.934820] LNet: Removed LNI 192.168.206.40@tcp [ 1560.964722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.063484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.096322] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1561.733597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1561.845864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1562.123714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.174485] LNet: Removed LNI 192.168.206.40@tcp [ 1563.214221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.302898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1563.353780] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1563.997717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1564.300278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.356156] LNet: Removed LNI 192.168.206.40@tcp [ 1565.401928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.458974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.475120] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1566.117905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1566.184553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.239505] LNet: Removed LNI 192.168.206.40@tcp [ 1566.282273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.333460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.351773] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1567.001541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1567.075577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.107877] LNet: Removed LNI 192.168.206.40@tcp [ 1567.147533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.209288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1567.319757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.362317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.437982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1567.523777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.578710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.635063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1567.708888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.743086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.795274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1567.827553] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1568.712877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.755266] LNet: Removed LNI 192.168.206.40@tcp [ 1568.782556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.851763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1575.632392] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:56:07 (1788569767) [ 1575.790801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.830514] LNet: 85562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.832797] LNet: 85562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1575.882594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.940773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1575.970188] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/180] [ 1577.185142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.204654] LNetError: 85703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.212160] LNetError: 85703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1577.230976] LNet: Removed LNI 192.168.206.40@tcp231 [ 1577.259701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.376438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1577.418825] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/180] [ 1577.424806] LNet: 85711:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.881319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1577.922853] LNet: Removed LNI 192.168.206.40@tcp231 [ 1577.940265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1577.973569] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/180] [ 1579.044368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.065479] LNet: Removed LNI 192.168.206.40@tcp231 [ 1579.076687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.126872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1579.143458] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/0] [ 1579.151695] LNet: 85864:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.597489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1579.640348] LNet: Removed LNI 192.168.206.40@tcp231 [ 1579.661773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1579.706970] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/0] [ 1580.718124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.744610] LNet: Removed LNI 192.168.206.40@tcp231 [ 1580.777299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.832135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1580.856325] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/60] [ 1580.860418] LNet: 86015:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.175036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1581.207147] LNet: Removed LNI 192.168.206.40@tcp231 [ 1581.228174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1581.249990] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/60] [ 1582.334840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.364879] LNet: Removed LNI 192.168.206.40@tcp231 [ 1582.385784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.452993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1582.475141] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/180] [ 1582.886530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1582.921748] LNet: Removed LNI 192.168.206.40@tcp231 [ 1582.933988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1582.966074] LNet: Added LNI 192.168.206.40@tcp231 [8/256/0/180] [ 1589.930770] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:56:21 (1788569781) [ 1590.087824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.139078] LNet: Removed LNI 192.168.206.40@tcp231 [ 1590.192248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.275607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1590.304583] LNet: Added LNI 192.168.206.40@tcp232 [8/256/0/180] [ 1591.147985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1591.193764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1591.229606] LNet: Removed LNI 192.168.206.40@tcp232 [ 1591.255637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1591.289272] LNet: Added LNI 192.168.206.40@tcp232 [8/256/0/180] [ 1592.184247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1592.224147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1592.248707] LNet: Removed LNI 192.168.206.40@tcp232 [ 1592.262758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1592.327342] LNet: Added LNI 192.168.206.40@tcp232 [8/256/0/180] [ 1593.180436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1593.237725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1593.273673] LNet: Removed LNI 192.168.206.40@tcp232 [ 1593.296467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1593.333530] LNet: Added LNI 192.168.206.40@tcp232 [8/256/0/180] [ 1594.011666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1594.054241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1594.090701] LNet: Removed LNI 192.168.206.40@tcp232 [ 1594.102201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1600.929718] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:56:32 (1788569792) [ 1601.864824] Key type .llcrypt unregistered [ 1601.867879] Key type ._llcrypt unregistered [ 1608.366905] Key type ._llcrypt registered [ 1608.370761] Key type .llcrypt registered [ 1608.683737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1608.712712] alg: No test for adler32 (adler32-zlib) [ 1609.603921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1609.636338] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1618.673978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1627.829165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.937331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1627.994750] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1629.664561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_interrupt -r 2 [ 1629.707652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_interrupt -r 2 [ 1629.740651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_interrupt -r 2 [ 1629.779336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_interrupt -r 2 [ 1629.826710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_interrupt -r 2 [ 1629.865629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_interrupt -r 2 [ 1629.892454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_interrupt -r 2 [ 1629.947820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_interrupt -r 2 [ 1629.986674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1630.007833] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1630.014698] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1630.295137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1630.330544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1630.369572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1630.418954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1630.451951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1630.491213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1630.554815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1630.612706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1630.653693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1630.696522] LNet: 88666:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1630.732513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1630.785383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1630.826446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1630.873957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.912990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1630.965213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1631.428441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_dropped -r 2 [ 1631.481586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_dropped -r 2 [ 1631.532883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_dropped -r 2 [ 1631.594497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_dropped -r 2 [ 1631.641851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_dropped -r 2 [ 1631.713278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_dropped -r 2 [ 1631.798666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_dropped -r 2 [ 1631.847537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_dropped -r 2 [ 1631.902922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1631.928481] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1632.154293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1632.187685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1632.211390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1632.241841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1632.269288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1632.313058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1632.355974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1632.408638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1632.448265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1632.501363] LNet: 88788:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1632.534735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1632.608461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.645859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.711724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.751522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.808339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1633.364245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_aborted -r 2 [ 1633.416600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_aborted -r 2 [ 1633.461198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_aborted -r 2 [ 1633.509435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_aborted -r 2 [ 1633.545954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_aborted -r 2 [ 1633.581964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_aborted -r 2 [ 1633.631289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_aborted -r 2 [ 1633.667938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_aborted -r 2 [ 1633.719168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1633.743398] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1634.085628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1634.111201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1634.144900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1634.186843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1634.250892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1634.300628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1634.332715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1634.370878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1634.393238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1634.406484] LNet: 88910:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1634.420805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1634.448579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1634.478546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1634.500969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1634.525987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1634.560275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.208336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_no_route -r 2 [ 1635.265020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_no_route -r 2 [ 1635.315238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_no_route -r 2 [ 1635.353312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_no_route -r 2 [ 1635.420832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_no_route -r 2 [ 1635.494084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_no_route -r 2 [ 1635.536496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_no_route -r 2 [ 1635.613302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_no_route -r 2 [ 1635.689617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1636.050573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.086954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1636.131550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.166839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.212083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1636.249235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.293879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.333982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1636.391489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1636.412188] LNet: 89033:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1636.439883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1636.471433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1636.498988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.525971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.551421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.578264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1637.021161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e local_timeout -r 2 [ 1637.050549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e local_timeout -r 2 [ 1637.082620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e local_timeout -r 2 [ 1637.110267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e local_timeout -r 2 [ 1637.138971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e local_timeout -r 2 [ 1637.170357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e local_timeout -r 2 [ 1637.206558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e local_timeout -r 2 [ 1637.242641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e local_timeout -r 2 [ 1637.272203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1637.295524] LNet: There was an unexpected network error while writing to 192.168.206.140: rc = -22 [ 1637.299495] LNet: Skipped 1 previous similar message [ 1637.596744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1637.649841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1637.676396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1637.702468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1637.728880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1637.754853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1637.785858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1637.831156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1637.865284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1637.906318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1637.967524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.016384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.049151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1638.083660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1638.117749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1638.612571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp -m GET -e remote_dropped -r 2 [ 1638.671286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.140@tcp1 -m GET -e remote_dropped -r 2 [ 1638.714411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp -m GET -e remote_dropped -r 2 [ 1638.765794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp -d 192.168.206.40@tcp1 -m GET -e remote_dropped -r 2 [ 1638.816816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp -m GET -e remote_dropped -r 2 [ 1638.856988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.140@tcp1 -m GET -e remote_dropped -r 2 [ 1638.916569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp -m GET -e remote_dropped -r 2 [ 1638.967737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.40@tcp1 -d 192.168.206.40@tcp1 -m GET -e remote_dropped -r 2 [ 1639.005860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.40@tcp 192.168.206.140@tcp [ 1639.041559] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.140@tcp [ 1639.360461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1639.393445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1639.424802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1639.454414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1639.495488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1639.541332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1639.573109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1639.603865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1639.637958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1639.677088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1639.724324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1639.750777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1639.782987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1639.831447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.883866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.874697] LNet: 89351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.883360] LNetError: 89351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.891815] LNet: Removed LNI 192.168.206.40@tcp [ 1645.904985] LNet: Removed LNI 192.168.206.40@tcp1 [ 1646.362598] Key type .llcrypt unregistered [ 1646.369248] Key type ._llcrypt unregistered [ 1664.220581] Key type ._llcrypt registered [ 1664.223353] Key type .llcrypt registered [ 1665.787496] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:57:37 (1788569857) [ 1666.122249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.140348] alg: No test for adler32 (adler32-zlib) [ 1667.103612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.189897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.223773] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1668.176472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1668.207978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.40@tcp [ 1668.231723] LNetError: 90326:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.40@tcp to recovery [ 1668.374514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.40@tcp [ 1668.524993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1675.541376] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:57:46 (1788569866) [ 1675.941368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.960792] LNet: 90916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.965399] LNetError: 90916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.982457] LNet: Removed LNI 192.168.206.40@tcp [ 1676.018059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.104190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.140787] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1677.065883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1677.115835] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1678.440281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.40@tcp [ 1678.494718] LNetError: 91183:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.40@tcp to recovery [ 1679.135757] LNet: 1 local NIs in recovery (showing 1): 192.168.206.40@tcp [ 1679.139487] LNet: There was an unexpected network error while writing to 192.168.206.40: rc = -22 [ 1679.782323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.40@tcp [ 1689.048307] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:58:00 (1788569880) [ 1691.196587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg640-server [ 1692.892614] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:58:04 (1788569884) [ 1694.842797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg640-server [ 1696.393826] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:58:08 (1788569888) [ 1698.011822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg640-server [ 1699.867782] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:58:11 (1788569891) [ 1700.117669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.164163] LNet: 92813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.174354] LNetError: 92813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.187165] LNet: Removed LNI 192.168.206.40@tcp [ 1700.210938] LNet: Removed LNI 192.168.206.40@tcp1 [ 1700.242694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.296494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.321441] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1701.059616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1701.128585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1701.174223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1701.305638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1701.361359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1701.502641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1708.165052] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:58:19 (1788569899) [ 1708.356699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.392791] LNet: 93544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.402616] LNetError: 93544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.412378] LNetError: 93544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1708.441499] LNet: Removed LNI 192.168.206.40@tcp [ 1708.479985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.564053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.591179] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1715.293725] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:58:26 (1788569906) [ 1715.429674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.459869] LNet: 94245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.465651] LNetError: 94245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.487354] LNet: Removed LNI 192.168.206.40@tcp [ 1715.504625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.608398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.649401] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1722.282616] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:58:33 (1788569913) [ 1723.864948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1725.670506] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:58:37 (1788569917) [ 1726.139753] LNet: 95281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.151899] LNetError: 95281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.173958] LNet: Removed LNI 192.168.206.40@tcp [ 1726.851828] Key type .llcrypt unregistered [ 1726.857513] Key type ._llcrypt unregistered [ 1733.568400] Key type ._llcrypt registered [ 1733.570569] Key type .llcrypt registered [ 1733.761654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.801977] alg: No test for adler32 (adler32-zlib) [ 1734.758436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1734.803624] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1743.360442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1748.764851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.140@tcp [ 1749.205699] LNet: 95696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.217623] LNetError: 95696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.249382] LNet: Removed LNI 192.168.206.40@tcp [ 1749.722418] Key type .llcrypt unregistered [ 1749.725408] Key type ._llcrypt unregistered [ 1762.631403] Key type ._llcrypt registered [ 1762.644078] Key type .llcrypt registered [ 1764.281645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1766.029562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1768.041888] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:59:19 (1788569959) [ 1769.512914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1771.497714] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:59:22 (1788569962) [ 1773.284576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg640-server [ 1775.293660] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:59:26 (1788569966) [ 1777.158273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg640-server [ 1778.895906] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:59:30 (1788569970) [ 1780.577126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.605935] alg: No test for adler32 (adler32-zlib) [ 1781.795863] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1782.157159] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1783.823689] Key type lgssc registered [ 1785.178391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.032856] Key type lgssc unregistered [ 1805.429801] LNet: 98868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.441882] LNetError: 98868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.462802] LNet: Removed LNI 192.168.206.40@tcp [ 1806.272963] Key type .llcrypt unregistered [ 1806.275654] Key type ._llcrypt unregistered [ 1820.909803] Key type ._llcrypt registered [ 1820.912391] Key type .llcrypt registered [ 1822.158204] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:00:13 (1788570013) [ 1822.658606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.668185] alg: No test for adler32 (adler32-zlib) [ 1823.589536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.681244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.749364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.793962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.846297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1829.928988] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:00:21 (1788570021) [ 1830.035165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.079779] LNet: 100276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.120802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.194468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.216453] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1831.018316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.038332] LNet: 100425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.044643] LNetError: 100425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.057956] LNet: Removed LNI 192.168.206.40@tcp [ 1831.078582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.127546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1831.145655] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1831.637753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.659458] LNetError: 100453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.671520] LNet: Removed LNI 192.168.206.40@tcp [ 1831.689575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.728560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1831.745441] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1837.757584] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:00:29 (1788570029) [ 1837.878459] LNet: 101036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.885331] LNet: 101036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1837.911870] LNetError: 101036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.943544] LNet: Removed LNI 192.168.206.40@tcp [ 1838.677529] Key type .llcrypt unregistered [ 1838.685651] Key type ._llcrypt unregistered [ 1845.490911] Key type ._llcrypt registered [ 1845.494548] Key type .llcrypt registered [ 1845.744755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1845.779797] alg: No test for adler32 (adler32-zlib) [ 1846.797161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.863202] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1846.867058] LNet: *** cfs_fail_loc=e003, val=100*** [ 1847.796308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1847.827402] LNet: *** cfs_fail_loc=e003, val=100*** [ 1847.829988] LNet: Skipped 2 previous similar messages [ 1847.854501] Ping reply received with mapping: [ 1847.858584] found metadata reply NI[192.168.206.40@tcp] [ 1847.862896] 28 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1847.867842] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1847.870498] 5a 5a 5a 5a 5a 5a 5a 5a [ 1848.164841] LNet: 101438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.188751] LNetError: 101438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.212372] LNet: Removed LNI 192.168.206.40@tcp [ 1848.889493] Key type .llcrypt unregistered [ 1848.892878] Key type ._llcrypt unregistered [ 1860.723625] Key type ._llcrypt registered [ 1860.725108] Key type .llcrypt registered [ 1862.257760] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:00:53 (1788570053) [ 1862.687244] Key type .llcrypt unregistered [ 1862.699215] Key type ._llcrypt unregistered [ 1869.175699] Key type ._llcrypt registered [ 1869.190279] Key type .llcrypt registered [ 1869.530286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1869.554664] alg: No test for adler32 (adler32-zlib) [ 1870.610132] LNetError: 102488:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1870.910417] Key type .llcrypt unregistered [ 1870.913248] Key type ._llcrypt unregistered [ 1876.121196] Key type ._llcrypt registered [ 1876.127299] Key type .llcrypt registered [ 1877.443943] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:01:08 (1788570068) [ 1878.164359] Key type .llcrypt unregistered [ 1878.166920] Key type ._llcrypt unregistered [ 1884.924585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1884.928738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1887.508376] Key type ._llcrypt registered [ 1887.511748] Key type .llcrypt registered [ 1887.690219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1887.724991] alg: No test for adler32 (adler32-zlib) [ 1888.662786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.711500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1897.804799] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:01:29 (1788570089) [ 1897.993700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.033623] LNet: 104260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.048461] LNetError: 104260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1898.070880] LNet: Removed LNI 10.1.2.3@tcp [ 1898.111064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.840556] Key type .llcrypt unregistered [ 1898.847144] Key type ._llcrypt unregistered [ 1905.444855] Key type ._llcrypt registered [ 1905.446110] Key type .llcrypt registered [ 1905.684479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1905.713561] alg: No test for adler32 (adler32-zlib) [ 1906.634225] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1906.703378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1906.849712] LNet: 104548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.857519] LNetError: 104548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.874768] LNet: Removed LNI 192.168.206.40@tcp [ 1907.494787] Key type .llcrypt unregistered [ 1907.497847] Key type ._llcrypt unregistered [ 1920.014176] Key type ._llcrypt registered [ 1920.016702] Key type .llcrypt registered [ 1921.400544] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:01:52 (1788570112) [ 1922.113322] Key type .llcrypt unregistered [ 1922.115284] Key type ._llcrypt unregistered [ 1928.906413] Key type ._llcrypt registered [ 1928.908711] Key type .llcrypt registered [ 1929.107777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1929.123584] alg: No test for adler32 (adler32-zlib) [ 1936.021765] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:02:07 (1788570127) [ 1936.167896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.206214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.258821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1936.292410] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1937.046291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.091117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1937.128404] LNet: Removed LNI 192.168.206.40@tcp [ 1937.356513] LNet: 106289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.720537] Key type .llcrypt unregistered [ 1937.723883] Key type ._llcrypt unregistered [ 1947.988634] Key type ._llcrypt registered [ 1947.990743] Key type .llcrypt registered [ 1949.258918] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:02:20 (1788570140) [ 1949.682853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.693568] alg: No test for adler32 (adler32-zlib) [ 1950.606425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.681104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.701832] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1951.441886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 1951.479637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.40@tcp [ 1951.494280] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1951.496670] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1951.500994] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1951, 0, 0 [ 1951.507815] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1951.509888] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1951.513986] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1951.516398] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1951, 0, 0 [ 1951.521782] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1951.524625] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1951.527879] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1951.531369] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1951, 0, 2001 [ 1951.535342] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1951.537798] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1951.541345] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1951.544765] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1951, 0, 2001 [ 1951.550640] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1958.066417] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:02:29 (1788570149) [ 1958.752473] LNet: 107826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.758363] LNetError: 107826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.782599] LNet: Removed LNI 192.168.206.40@tcp [ 1960.615520] Key type .llcrypt unregistered [ 1960.618702] Key type ._llcrypt unregistered [ 1966.989063] Key type ._llcrypt registered [ 1966.993852] Key type .llcrypt registered [ 1967.332113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.350469] alg: No test for adler32 (adler32-zlib) [ 1968.277052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1968.303469] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 1976.264575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 1985.385690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.441607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1985.469852] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 1986.588796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1986.800137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1986.804954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1986.812373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1989.122980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1989.156171] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2000.143986] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:03:11 (1788570191) [ 2000.397921] LNet: 109331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.406656] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.414584] LNet: Removed LNI 192.168.206.40@tcp [ 2001.439525] LNet: Removed LNI 192.168.206.40@tcp1 [ 2001.443389] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2001.450198] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2001.476072] LNet: Removed LNI 10.1.2.3@tcp99 [ 2002.100751] Key type .llcrypt unregistered [ 2002.102598] Key type ._llcrypt unregistered [ 2007.802921] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2007.812125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2010.453726] Key type ._llcrypt registered [ 2010.455958] Key type .llcrypt registered [ 2010.698978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.723900] alg: No test for adler32 (adler32-zlib) [ 2011.686793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.766153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2011.802865] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2012.827618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2012.874971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2015.607430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2015.666391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2015.733489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2015.754641] LNetError: 110072:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2015.793558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2015.835631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2016.151539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.205599] LNet: 110092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.209573] LNetError: 110092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2016.212821] LNet: Removed LNI 10.1.2.3@tcp [ 2017.256738] LNet: Removed LNI 192.168.206.40@tcp [ 2018.335870] Key type .llcrypt unregistered [ 2018.338875] Key type ._llcrypt unregistered [ 2030.936775] Key type ._llcrypt registered [ 2030.938967] Key type .llcrypt registered [ 2032.324404] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:03:43 (1788570223) [ 2033.043101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.061443] alg: No test for adler32 (adler32-zlib) [ 2034.097747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.195537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.231417] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2035.117440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2035.208294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg640-client@tcp [ 2041.701375] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:03:53 (1788570233) [ 2041.849592] LNet: 111621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.854641] LNetError: 111621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.871268] LNet: Removed LNI 192.168.206.40@tcp [ 2042.791442] Key type .llcrypt unregistered [ 2042.793825] Key type ._llcrypt unregistered [ 2050.126259] Key type ._llcrypt registered [ 2050.132323] Key type .llcrypt registered [ 2050.494470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.516485] alg: No test for adler32 (adler32-zlib) [ 2051.429542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2051.462928] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2051.484395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2058.249714] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:04:09 (1788570249) [ 2059.844929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2061.576471] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:04:13 (1788570253) [ 2063.133754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2065.205678] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:04:16 (1788570256) [ 2065.295561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.339619] LNet: 113040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.344780] LNetError: 113040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.372130] LNet: Removed LNI 192.168.206.40@tcp [ 2065.443159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.505553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.542249] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2066.730862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2066.824270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2074.384934] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:04:25 (1788570265) [ 2074.492878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2074.540791] LNet: 113751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.546085] LNetError: 113751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.571143] LNet: Removed LNI 192.168.206.40@tcp [ 2074.618762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.676656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2074.703590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2075.324430] Key type .llcrypt unregistered [ 2075.328207] Key type ._llcrypt unregistered [ 2087.029779] Key type ._llcrypt registered [ 2087.031487] Key type .llcrypt registered [ 2088.295550] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:04:39 (1788570279) [ 2088.767564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.788621] alg: No test for adler32 (adler32-zlib) [ 2089.712034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.768847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2089.803822] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2090.504727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2090.582385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2097.770462] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:04:49 (1788570289) [ 2097.854550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.894951] LNet: 115302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.905025] LNetError: 115302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.933527] LNet: Removed LNI 192.168.206.40@tcp [ 2097.953569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.066132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2098.129487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2104.795143] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:04:56 (1788570296) [ 2105.044861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.105110] LNet: 115890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.128403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.174892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2105.496412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2105.529387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.586538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2106.088519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2106.140414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2106.217477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.251436] LNet: 115923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.280971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.355179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2106.757893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2106.789903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2106.891881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2107.477803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2107.505911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2113.817532] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:05:05 (1788570305) [ 2113.905374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.942781] LNet: 116517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.982136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.087264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.153426] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2115.064115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2115.086534] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2115.981966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2116.057140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2116.331859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.392475] LNetError: 116803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.405275] LNet: Removed LNI 192.168.206.40@tcp [ 2116.424733] LNet: Removed LNI 192.168.206.40@tcp2 [ 2116.442323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.517586] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2116.524990] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2116.529152] LNet: 116809:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.142392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2117.355676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.399273] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.411337] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2117.419769] LNet: Removed LNI 192.168.206.40@tcp [ 2117.445811] LNet: Removed LNI 192.168.206.40@tcp2 [ 2117.494732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.592153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2117.627533] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2118.599534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2118.635582] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2119.609135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2119.643717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2119.707641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2119.902877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.931607] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.937075] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2119.941924] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.946972] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2119.951989] LNet: Removed LNI 192.168.206.40@tcp [ 2120.998929] LNet: Removed LNI 192.168.206.40@tcp2 [ 2121.051315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.134743] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2121.141456] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2121.147773] LNet: 117136:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2121.878438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2122.062954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.078409] LNet: Removed LNI 192.168.206.40@tcp [ 2123.113568] LNet: Removed LNI 192.168.206.40@tcp2 [ 2123.152260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.236363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.272217] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2124.201286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.218280] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2125.031678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2125.095856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2125.351265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.385812] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.397123] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2125.405818] LNet: Removed LNI 192.168.206.40@tcp [ 2125.419798] LNet: Removed LNI 192.168.206.40@tcp2 [ 2125.473360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.539506] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2125.545626] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2125.551187] LNet: 117459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2126.221158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2126.503658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.583148] LNet: Removed LNI 192.168.206.40@tcp [ 2127.601224] LNet: Removed LNI 192.168.206.40@tcp2 [ 2127.649377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.710409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.739125] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2128.627829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2128.650479] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2129.559857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2129.598441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2129.693813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2129.949829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.974720] LNet: 117781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.978715] LNet: 117781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2131.039280] LNet: Removed LNI 192.168.206.40@tcp [ 2132.086448] LNet: Removed LNI 192.168.206.40@tcp2 [ 2132.129164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.232459] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2132.239947] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2132.249409] LNet: 117788:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.822552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2133.013376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.030319] LNet: Removed LNI 192.168.206.40@tcp [ 2134.054631] LNet: Removed LNI 192.168.206.40@tcp2 [ 2134.085090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.145953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.179602] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2135.047630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.079420] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2136.035676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.076710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2136.162915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2136.598582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.638855] LNetError: 118122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.648816] LNetError: 118122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2136.669308] LNet: Removed LNI 192.168.206.40@tcp [ 2136.684488] LNet: Removed LNI 192.168.206.40@tcp2 [ 2136.710771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.774307] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2136.778655] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2136.785702] LNet: 118128:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.531407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2137.996105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.041594] LNet: Removed LNI 192.168.206.40@tcp [ 2139.063418] LNet: Removed LNI 192.168.206.40@tcp2 [ 2139.110264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.184383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.204792] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2139.976728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2139.996342] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2140.844489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2140.887548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2140.915859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2140.983580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2141.392655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.431124] LNet: Removed LNI 192.168.206.40@tcp [ 2143.461433] LNet: Removed LNI 192.168.206.40@tcp2 [ 2143.506149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.595510] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2143.604776] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2144.216026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2151.500159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2153.487429] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:05:44 (1788570344) [ 2153.593854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.618661] LNet: 119242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.635693] LNet: 119242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2153.656318] LNetError: 119242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.669118] LNetError: 119242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2153.679848] LNet: Removed LNI 192.168.206.40@tcp [ 2154.727855] LNet: Removed LNI 192.168.206.40@tcp2 [ 2154.774386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.129366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.177039] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2156.014496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2156.030378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2156.878322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.912234] LNet: Removed LNI 10.1.2.3@tcp [ 2156.930128] LNet: Removed LNI 192.168.206.40@tcp [ 2156.963199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.044897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2157.087821] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2157.110968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.697772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2157.717796] LNet: Removed LNI 192.168.206.40@tcp [ 2157.735158] LNet: Removed LNI 10.1.2.3@tcp [ 2164.351455] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:05:55 (1788570355) [ 2164.633953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.679729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.733833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2164.754201] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2165.392917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2165.447290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2165.517615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.569281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.611777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.663341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.749289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.797567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.853432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.904525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.939894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2165.984323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2166.154621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2172.555604] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 21:06:04 (1788570364) [ 2172.631080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.689896] LNet: Removed LNI 192.168.206.40@tcp [ 2173.724415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.804199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.847530] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2174.730131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2174.778557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2174.848592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2174.909338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2174.971859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.045339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.076476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.136954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.173744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.224680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.266716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.322944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.504959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2175.564030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.604874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.650596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.692651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.730341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.784504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.829476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.898090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.939566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2175.989557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.40@tcp [ 2182.883869] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:06:14 (1788570374) [ 2183.088429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.175590] LNet: Removed LNI 192.168.206.40@tcp [ 2184.245533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.411808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2184.650622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.662114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.670697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2187.094409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.118551] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2187.856921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2187.877161] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2189.977692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2190.060044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2190.170606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.40@tcp -m PUT -e local_timeout -r 1 [ 2190.287638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2190.349696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2190.354817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2190.559953] LNetError: 121683:0:(lib-move.c:4794:lnet_parse()) 192.168.206.40@tcp, src 192.168.206.40@tcp: Dropping PUT (error -108 looking up sender) [ 2191.583626] LNet: 122143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.593338] LNet: 122143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2191.609368] LNetError: 122143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2191.614443] LNetError: 122143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2191.624042] LNet: Removed LNI 10.1.2.3@tcp [ 2192.690228] LNet: Removed LNI 192.168.206.40@tcp [ 2193.097850] Key type .llcrypt unregistered [ 2193.100463] Key type ._llcrypt unregistered [ 2205.372535] Key type ._llcrypt registered [ 2205.374428] Key type .llcrypt registered [ 2206.605827] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:06:38 (1788570398) [ 2207.293551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.309587] alg: No test for adler32 (adler32-zlib) [ 2208.272130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.801728] LNet: 122994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.785577] Key type .llcrypt unregistered [ 2209.789625] Key type ._llcrypt unregistered [ 2218.159681] Key type ._llcrypt registered [ 2218.163721] Key type .llcrypt registered [ 2218.372924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2218.386434] alg: No test for adler32 (adler32-zlib) [ 2219.340128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2219.386722] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2227.606462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 2227.790303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2228.457396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.140@tcp [ 2228.639463] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.140@tcp [ 2228.704574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2228.736530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2228.776751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2228.803870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2228.826758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2228.855873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2228.892864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2228.926551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2228.958915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2228.983943] LNet: 123443:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2229.008620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2229.071779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2229.113841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2229.142591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2229.182121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2229.212792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2229.880778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2230.194370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2230.233345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2230.269847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2230.325794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2230.354667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2230.386511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2230.410143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2230.459411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2230.510651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2230.530081] LNet: 123563:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.549403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2230.598152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2230.637771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2230.666188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2230.696500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2230.754143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2231.488884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.140@tcp [ 2231.748262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2231.774362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2231.827884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2231.867901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2231.940886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2231.974671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2232.021474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2232.073554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2232.138447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2232.159512] LNet: 123682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2232.179371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2232.219854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2232.268414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2232.335609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2232.374219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2232.404767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2232.998925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2233.343700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2233.386305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2233.441573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2233.485662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2233.537991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2233.561738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2233.592733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2233.625904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2233.655631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2233.690535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2233.738656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2233.771319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2233.812403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2233.852355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2233.907323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2234.179202] LNet: 123847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.184704] LNetError: 123847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.236890] LNet: Removed LNI 192.168.206.40@tcp [ 2235.579736] Key type .llcrypt unregistered [ 2235.582720] Key type ._llcrypt unregistered [ 2253.568747] Key type ._llcrypt registered [ 2253.570416] Key type .llcrypt registered [ 2255.189375] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:07:26 (1788570446) [ 2255.748264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.807996] alg: No test for adler32 (adler32-zlib) [ 2256.773973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.213498] LNet: 124733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.942713] Key type .llcrypt unregistered [ 2257.943946] Key type ._llcrypt unregistered [ 2265.239296] Key type ._llcrypt registered [ 2265.244499] Key type .llcrypt registered [ 2265.470757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2265.491502] alg: No test for adler32 (adler32-zlib) [ 2266.470507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2266.506905] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2274.693191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.140@tcp [ 2284.148505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.208742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2284.239048] LNet: Added LNI 192.168.206.40@tcp1 [8/256/0/180] [ 2285.176146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2285.720256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.140@tcp [ 2285.985516] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.140@tcp [ 2286.084291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.136424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.173170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.200623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.228140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2286.289139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.336989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.408520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.474026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.504521] LNet: 125386:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.545441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.589449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.640689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2286.695598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2286.778447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2286.859250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.796241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2288.077993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.117890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.159529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.193863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.265683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2288.291281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.316734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.361979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.409675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.443687] LNet: 125505:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.465967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.494477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.538088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.582603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.623349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.671991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.419352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.140@tcp [ 2289.646260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.679111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2289.705220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.746724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.788069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2289.825332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.857171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.901847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.942912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.969857] LNet: 125624:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2289.997960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2290.052439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2290.115412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2290.184122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.246043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.300903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.920590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2291.159404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.197940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2291.229499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.258438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.295351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2291.325425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.365723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.406226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.456997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.518269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.580112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.625641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.692983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.728977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.768046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2296.454564] LNet: 125825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.461302] LNetError: 125825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.469971] LNet: Removed LNI 192.168.206.40@tcp [ 2296.480175] LNet: Removed LNI 192.168.206.40@tcp1 [ 2296.894657] Key type .llcrypt unregistered [ 2296.896241] Key type ._llcrypt unregistered [ 2311.939319] Key type ._llcrypt registered [ 2311.941250] Key type .llcrypt registered [ 2313.072428] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:08:24 (1788570504) [ 2313.491306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.531175] alg: No test for adler32 (adler32-zlib) [ 2314.418290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.456278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2314.477589] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2315.143335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2315.167950] LNet: Added LNI 192.168.206.40@tcp2 [8/256/0/180] [ 2316.027370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.40@tcp [ 2316.106441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2317.151434] LNet: Removed LNI 192.168.206.40@tcp [ 2323.604328] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:08:35 (1788570515) [ 2325.058537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2327.039759] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:08:38 (1788570518) [ 2334.442267] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:08:45 (1788570525) [ 2341.519815] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:08:53 (1788570533) [ 2349.272308] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:09:00 (1788570540) [ 2356.664505] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:09:08 (1788570548) [ 2364.379823] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:09:15 (1788570555) [ 2371.472694] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:09:23 (1788570563) [ 2378.969832] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:09:30 (1788570570) [ 2386.048840] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:09:37 (1788570577) [ 2392.800789] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:09:44 (1788570584) [ 2399.375797] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:09:51 (1788570591) [ 2406.298776] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:09:58 (1788570598) [ 2412.827791] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:10:04 (1788570604) [ 2419.426836] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:10:10 (1788570610) [ 2426.911907] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:10:18 (1788570618) [ 2433.296328] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:10:24 (1788570624) [ 2442.206357] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:10:32 (1788570632) [ 2451.172258] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:10:42 (1788570642) [ 2451.397724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.439226] LNet: 137515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.447664] LNetError: 137515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.485665] LNet: Removed LNI 192.168.206.40@tcp2 [ 2451.575762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.769259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.809729] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2453.451276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.533660] LNet: 137661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.548514] LNetError: 137661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2453.578758] LNet: Removed LNI 192.168.206.40@tcp [ 2453.605493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.756725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2453.818604] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2460.551333] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:10:52 (1788570652) [ 2467.057426] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:10:58 (1788570658) [ 2473.146277] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:11:04 (1788570664) [ 2479.844799] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:11:11 (1788570671) [ 2486.105919] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:11:17 (1788570677) [ 2491.993883] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:11:23 (1788570683) [ 2499.020744] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:11:30 (1788570690) [ 2506.858214] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:11:38 (1788570698) [ 2514.238939] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:11:45 (1788570705) [ 2521.564850] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:11:53 (1788570713) [ 2528.640584] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:12:00 (1788570720) [ 2535.217242] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:12:06 (1788570726) [ 2541.857068] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2284 sec ========== 21:12:13 (1788570733) [ 2542.260978] LNet: 145124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.275049] LNetError: 145124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.294279] LNet: Removed LNI 192.168.206.40@tcp [ 2543.018768] Key type .llcrypt unregistered [ 2543.024611] Key type ._llcrypt unregistered [ 2555.709128] Key type ._llcrypt registered [ 2555.715098] Key type .llcrypt registered [ 2556.187704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2556.197758] alg: No test for adler32 (adler32-zlib) [ 2557.294703] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2557.517299] LNet: Added LNI 192.168.206.40@tcp [8/256/0/180] [ 2559.207162] Key type lgssc registered [ 2560.279701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2619.639379] Lustre: Mounted lustre-client - version 2.17.58_2_gfa0a981 [ 2624.641378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2631.429820] Lustre: Unmounted lustre-client [ 2659.386906] Key type lgssc unregistered [ 2659.658863] LNet: 148069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.671799] LNetError: 148069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.688403] LNet: Removed LNI 192.168.206.40@tcp [ 2660.358131] Key type .llcrypt unregistered [ 2660.359608] Key type ._llcrypt unregistered