[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436296134 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5410-0x000f541f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5220 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D6F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C0B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BCB (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C7F 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D0F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D47 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c0b-0xbffe1c7e] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c0a] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c7f-0xbffe1d0e] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d0f-0xbffe1d46] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d47-0xbffe1d6e] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002011] Switched APIC routing to physical x2apic. [ 0.003013] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.009099] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012000] *** VALIDATE selinux *** [ 0.017330] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022026] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025125] *** VALIDATE tmpfs *** [ 0.027426] *** VALIDATE proc *** [ 0.028241] *** VALIDATE cgroup *** [ 0.029011] *** VALIDATE cgroup2 *** [ 0.031135] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.037768] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.039000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040019] ... version: 2 [ 0.041012] ... bit width: 48 [ 0.042011] ... generic registers: 4 [ 0.043013] ... value mask: 0000ffffffffffff [ 0.044016] ... max period: 00007fffffffffff [ 0.045013] ... fixed-purpose events: 3 [ 0.046012] ... event mask: 000000070000000f [ 0.047295] rcu: Hierarchical SRCU implementation. [ 0.049420] smp: Bringing up secondary CPUs ... [ 0.050589] x86: Booting SMP configuration: [ 0.051038] .... node #0, CPUs: #1 #2 #3 [ 0.055095] smp: Brought up 1 node, 4 CPUs [ 0.057015] smpboot: Max logical packages: 1 [ 0.058019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131450] node 0 deferred pages initialised in 73ms [ 0.135163] devtmpfs: initialized [ 0.136262] x86/mm: Memory block size: 128MB [ 0.139849] gcov: version magic: 0x41383552 [ 0.142225] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148333] pinctrl core: initialized pinctrl subsystem [ 0.150205] [ 0.150821] ************************************************************* [ 0.154018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.159016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161012] ** ** [ 0.164016] ** This means that this kernel is built to expose internal ** [ 0.165010] ** IOMMU data structures, which may compromise security on ** [ 0.168015] ** your system. ** [ 0.170011] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.175020] ** kernel, report this immediately to your vendor! ** [ 0.177009] ** ** [ 0.178008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181011] ************************************************************* [ 0.183497] NET: Registered protocol family 16 [ 0.185414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194106] cpuidle: using governor menu [ 0.196587] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199100] PCI: Using configuration type 1 for base access [ 0.201123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215031] cryptd: max_cpu_qlen set to 1000 [ 0.217226] ACPI: Added _OSI(Module Device) [ 0.219019] ACPI: Added _OSI(Processor Device) [ 0.221015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226545] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232584] ACPI: Interpreter enabled [ 0.234069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236016] ACPI: Using IOAPIC for interrupt routing [ 0.237097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249827] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264239] acpiphp: Slot [2] registered [ 0.265111] acpiphp: Slot [5] registered [ 0.267101] acpiphp: Slot [6] registered [ 0.268112] acpiphp: Slot [3] registered [ 0.270104] acpiphp: Slot [4] registered [ 0.271138] acpiphp: Slot [7] registered [ 0.273105] acpiphp: Slot [8] registered [ 0.274089] acpiphp: Slot [9] registered [ 0.275093] acpiphp: Slot [10] registered [ 0.277104] acpiphp: Slot [11] registered [ 0.278145] acpiphp: Slot [12] registered [ 0.281037] acpiphp: Slot [13] registered [ 0.282157] acpiphp: Slot [14] registered [ 0.284131] acpiphp: Slot [15] registered [ 0.285109] acpiphp: Slot [16] registered [ 0.287108] acpiphp: Slot [17] registered [ 0.288117] acpiphp: Slot [18] registered [ 0.290164] acpiphp: Slot [19] registered [ 0.291113] acpiphp: Slot [20] registered [ 0.292101] acpiphp: Slot [21] registered [ 0.294095] acpiphp: Slot [22] registered [ 0.295118] acpiphp: Slot [23] registered [ 0.296119] acpiphp: Slot [24] registered [ 0.297106] acpiphp: Slot [25] registered [ 0.299137] acpiphp: Slot [26] registered [ 0.300080] acpiphp: Slot [27] registered [ 0.301081] acpiphp: Slot [28] registered [ 0.302084] acpiphp: Slot [29] registered [ 0.303107] acpiphp: Slot [30] registered [ 0.304000] acpiphp: Slot [31] registered [ 0.304000] PCI host bridge to bus 0000:00 [ 0.305030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312021] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.314031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318228] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.331288] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.336028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340337] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343706] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.346048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.353020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.362985] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.368016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374443] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380034] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397018] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.405000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.439020] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.449000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451276] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.453380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.456376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.458258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.463568] iommu: Default domain type: Passthrough [ 0.465657] SCSI subsystem initialized [ 0.467174] ACPI: bus type USB registered [ 0.468142] usbcore: registered new interface driver usbfs [ 0.470105] usbcore: registered new interface driver hub [ 0.472136] usbcore: registered new device driver usb [ 0.474214] pps_core: LinuxPPS API ver. 1 registered [ 0.476014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.479094] PTP clock support registered [ 0.481160] EDAC MC: Ver: 3.0.0 [ 0.484143] PCI: Using ACPI for IRQ routing [ 0.485831] NetLabel: Initializing [ 0.487015] NetLabel: domain hash size = 128 [ 0.489014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.491112] NetLabel: unlabeled traffic allowed by default [ 0.493099] vgaarb: loaded [ 0.495315] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.497013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.508167] clocksource: Switched to clocksource kvm-clock [ 0.610583] VFS: Disk quotas dquot_6.6.0 [ 0.612245] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.614962] *** VALIDATE ramfs *** [ 0.615947] *** VALIDATE hugetlbfs *** [ 0.617288] pnp: PnP ACPI init [ 0.620563] pnp: PnP ACPI: found 6 devices [ 0.635562] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.638611] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.640964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.643137] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.645704] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.648171] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.651100] NET: Registered protocol family 2 [ 0.653552] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.658286] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.661551] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.666348] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.669764] TCP: Hash tables configured (established 65536 bind 65536) [ 0.672689] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.675404] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.678194] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.681042] NET: Registered protocol family 1 [ 0.684187] RPC: Registered named UNIX socket transport module. [ 0.687498] RPC: Registered udp transport module. [ 0.689317] RPC: Registered tcp transport module. [ 0.690850] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.693633] NET: Registered protocol family 44 [ 0.695436] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.697307] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.698899] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.700599] PCI: CLS 0 bytes, default 64 [ 0.702863] Unpacking initramfs... [ 2.126315] debug: unmapping init [mem 0xffff9bf5bcc64000-0xffff9bf5bffcffff] [ 2.129776] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.132020] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.134390] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.643177] Initialise system trusted keyrings [ 2.645026] Key type blacklist registered [ 2.647187] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.660110] zbud: loaded [ 2.669310] *** VALIDATE nfs *** [ 2.670742] *** VALIDATE nfs4 *** [ 2.676184] pstore: using deflate compression [ 2.679529] Platform Keyring initialized [ 2.786307] NET: Registered protocol family 38 [ 2.787871] Key type asymmetric registered [ 2.789356] Asymmetric key parser 'x509' registered [ 2.791768] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.794809] io scheduler mq-deadline registered [ 2.796750] io scheduler kyber registered [ 2.798566] io scheduler bfq registered [ 2.801354] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.804236] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.806958] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.809512] ACPI: Power Button [PWRF] [ 2.899301] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.990368] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.085541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.114968] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.145158] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.151249] Non-volatile memory driver v1.3 [ 3.153221] Linux agpgart interface v0.103 [ 3.193365] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.196584] vda: detected capacity change from 0 to 68530176 [ 3.212986] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.216192] vdb: detected capacity change from 0 to 1073741824 [ 3.223596] libphy: Fixed MDIO Bus: probed [ 3.231185] usbcore: registered new interface driver usbserial_generic [ 3.233958] usbserial: USB Serial support registered for generic [ 3.236565] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.241160] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.243035] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.245216] mousedev: PS/2 mouse device common for all mice [ 3.248560] rtc_cmos 00:05: RTC can wake from S4 [ 3.251233] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.255176] rtc_cmos 00:05: registered as rtc0 [ 3.257587] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.257968] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.261733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.263721] intel_pstate: CPU model not supported [ 3.270328] hid: raw HID events driver (C) Jiri Kosina [ 3.272328] usbcore: registered new interface driver usbhid [ 3.275040] usbhid: USB HID core driver [ 3.277194] drop_monitor: Initializing network drop monitor service [ 3.279844] Initializing XFRM netlink socket [ 3.282073] NET: Registered protocol family 10 [ 3.285233] Segment Routing with IPv6 [ 3.286864] NET: Registered protocol family 17 [ 3.289052] mpls_gso: MPLS GSO support [ 3.295250] RAS: Correctable Errors collector initialized. [ 3.297576] AVX version of gcm_enc/dec engaged. [ 3.299540] AES CTR mode by8 optimization enabled [ 3.367100] sched_clock: Marking stable (3367030961, 0)->(4291913180, -924882219) [ 3.371540] registered taskstats version 1 [ 3.374772] Loading compiled-in X.509 certificates [ 3.376981] zswap: loaded using pool lzo/zbud [ 3.402251] Key type big_key registered [ 3.413601] Key type encrypted registered [ 3.415578] ima: No TPM chip found, activating TPM-bypass! [ 3.417894] ima: Allocated hash algorithm: sha1 [ 3.419543] ima: No architecture policies found [ 3.421358] evm: Initialising EVM extended attributes: [ 3.423711] evm: security.selinux [ 3.425426] evm: security.ima [ 3.426633] evm: security.capability [ 3.428119] evm: HMAC attrs: 0x1 [ 3.432101] rtc_cmos 00:05: setting system clock to 2025-11-17 02:08:19 UTC (1763345299) [ 3.438766] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 3.441930] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 3.452133] Write protecting the kernel read-only data: 28672k [ 3.455621] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.458731] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 3.490322] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.499661] systemd[1]: Detected virtualization kvm. [ 3.501754] systemd[1]: Detected architecture x86-64. [ 3.503753] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.531544] systemd[1]: No hostname configured. [ 3.533251] systemd[1]: Set hostname to . [ 3.535131] random: systemd: uninitialized urandom read (16 bytes read) [ 3.537285] systemd[1]: Initializing machine ID from random generator. [ 3.603871] random: ln: uninitialized urandom read (6 bytes read) [ 3.685160] random: systemd: uninitialized urandom read (16 bytes read) [ 3.687595] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.691708] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.697698] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.373917] device-mapper: uevent: version 1.0.3 [ 4.376551] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.155181] random: fast init done [ 5.172063] virtio_net virtio0 ens2: renamed from eth0 [ 5.239482] scsi host0: ata_piix [ 5.255981] scsi host1: ata_piix [ 5.257640] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.260239] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.923717] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.935086] random: crng init done [ 9.936462] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.368499] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.436448] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.662809] SELinux: Disabled at runtime. [ 11.719202] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.729265] systemd[1]: Detected virtualization kvm. [ 11.731368] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.168617] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.172544] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.177803] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.182065] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.185252] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.192272] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.196453] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ 12.230371] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.583642] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.903112] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.907531] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.004581] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.021827] EDAC sbridge: Ver: 1.1.2 [ 14.090924] Key type dns_resolver registered [ 14.385352] NFS: Registering the id_resolver key type [ 14.387479] Key type id_resolver registered [ 14.389508] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg129-client login: [ 43.651238] libcfs: loading out-of-tree module taints kernel. [ 43.730865] Key type ._llcrypt registered [ 43.736893] Key type .llcrypt registered [ 43.930170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.938953] alg: No test for adler32 (adler32-zlib) [ 44.964678] Lustre: Lustre: Build Version: 2.16.61_45_g58940e0 [ 45.311562] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 46.935203] Key type lgssc registered [ 47.609711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.288029] hrtimer: interrupt took 6019302 ns [ 143.812870] Lustre: Mounted lustre-client [ 148.691356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 162.955819] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing check_logdir /tmp/testlogs/ [ 166.397796] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing yml_node [ 169.445325] Lustre: lustre-OST0000-osc-ffff9bf6054d8000: disconnect after 23s idle [ 170.174539] Lustre: DEBUG MARKER: Client: 2.16.61.45 [ 172.100775] Lustre: DEBUG MARKER: MDS: 2.16.61.45 [ 174.162258] Lustre: DEBUG MARKER: OSS: 2.16.61.45 [ 175.404444] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 21:11:10 EST 2025 [ 187.691112] LustreError: 7898:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9bf6054d8000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 187.713454] LustreError: 7898:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 187.751026] Lustre: Unmounted lustre-client [ 211.769494] Key type lgssc unregistered [ 212.061172] LNet: 8345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 212.072179] LNetError: 8345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 212.089175] LNet: Removed LNI 192.168.201.29@tcp [ 212.898152] Key type .llcrypt unregistered [ 212.901797] Key type ._llcrypt unregistered [ 231.290375] Key type ._llcrypt registered [ 231.291673] Key type .llcrypt registered [ 231.483178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 231.504140] alg: No test for adler32 (adler32-zlib) [ 232.400632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 232.470699] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 232.487914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 235.478327] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 235.572747] LNet: 9212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 235.581162] LNetError: 9212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 235.595132] LNet: Removed LNI 192.168.201.29@tcp [ 236.047498] Key type .llcrypt unregistered [ 236.049365] Key type ._llcrypt unregistered [ 240.004347] Key type ._llcrypt registered [ 240.005807] Key type .llcrypt registered [ 240.984716] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:12:15 (1763345535) [ 241.225116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 241.234356] alg: No test for adler32 (adler32-zlib) [ 242.100529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.151172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.165977] LNet: 9624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 242.195732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.239896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 242.255826] LNet: 9631:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 247.049336] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:12:22 (1763345542) [ 247.122368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.140692] LNet: 10201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 247.156404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.228231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 252.178563] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:12:27 (1763345547) [ 252.271233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.289678] LNet: 10787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.319394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.412726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 257.224752] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:12:32 (1763345552) [ 257.357493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.403168] LNet: 11374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 257.443078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.529715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 262.510962] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:12:37 (1763345557) [ 262.680646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.711146] LNet: 11959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.734811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.788900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 263.121814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 267.973906] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:12:42 (1763345562) [ 268.037525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.090264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.171017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 268.482990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 268.834155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 273.867569] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:12:49 (1763345569) [ 273.959498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.979384] LNet: 13144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.989420] LNet: 13144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 274.015222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.080195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 279.259333] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:12:54 (1763345574) [ 279.413143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.445272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.501728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 279.530636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 279.821291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 279.860454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 280.279341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 280.320501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 280.696704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 280.740075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 281.013979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 281.041382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 281.430615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 281.489082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 281.964686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 282.024405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 282.461790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 282.485343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 282.861407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 282.899744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 283.256780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 283.297277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 283.603286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 283.632803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 283.929192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 283.961516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 284.328881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 284.359936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 290.287751] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:13:05 (1763345585) [ 290.478586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.531789] LNet: 14408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.546975] LNet: 14408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 290.582802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.666700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 290.695557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 296.911570] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:13:11 (1763345591) [ 297.086392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.146073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.244153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 297.275851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 303.481107] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:13:18 (1763345598) [ 303.545213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.608691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.739763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 303.778229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 310.190838] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:13:24 (1763345604) [ 310.323986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.408972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.535899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 310.610939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 317.254382] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:13:32 (1763345612) [ 317.382561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.469419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.561275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 317.632634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 324.135338] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:13:38 (1763345618) [ 324.326454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.352786] LNet: 17343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.355709] LNet: 17343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 324.386339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.517145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 324.555059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 330.073954] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:13:44 (1763345624) [ 330.200591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.245761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.365066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.698440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 335.829682] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:13:50 (1763345630) [ 335.931741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.960675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.126471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 336.301568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 341.289499] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:13:56 (1763345636) [ 341.424613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.476343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.575748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 341.791580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 347.088971] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:14:02 (1763345642) [ 347.328492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.355377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.725829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 347.936955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 352.883501] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:14:07 (1763345647) [ 353.067725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.095811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.173652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.221737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 358.421919] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:14:13 (1763345653) [ 358.487665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.519719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.629274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.688717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 363.997965] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:14:18 (1763345658) [ 364.112585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.152197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.236164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 369.394414] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:14:24 (1763345664) [ 369.531049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.570393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.654365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 375.235155] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:14:30 (1763345670) [ 375.384197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.429636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.542905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 375.998847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 383.524411] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:14:37 (1763345677) [ 383.656503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.724681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.857720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 383.941565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 391.304858] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:14:45 (1763345685) [ 391.383907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.400155] LNet: 23951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.402825] LNet: 23951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 391.437488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.506042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 391.532560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 397.998239] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:14:52 (1763345692) [ 398.185896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.219242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.287739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 398.314692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 403.966765] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:14:58 (1763345698) [ 404.122797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.156583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.210769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 404.252616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 409.890350] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:15:04 (1763345704) [ 409.986407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.036653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.118265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 410.195439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 410.218153] LNetError: 25730:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 410.259934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 410.301201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 415.634942] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:15:10 (1763345710) [ 415.768363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.831973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.901650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WnjACsmXVL [ 415.968334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WnjACsmXVL [ 416.044177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WnjACsmXVL [ 421.693577] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:15:16 (1763345716) [ 421.835677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.918192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.990892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 422.055099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 422.132851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 428.073227] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:15:22 (1763345722) [ 428.302956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.348379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.418802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 430.791593] LNet: Message forwarding will be enabled in 2s [ 434.911135] LNet: Message forwarding enabled [ 435.224263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 435.263280] LNet: Message forwarding will stop in 2s to 6s [ 439.632263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 440.132955] LNet: Message forwarding disabled [ 442.120458] LNet: Message forwarding will be enabled in 2s [ 445.280890] LNet: Message forwarding enabled [ 447.830696] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:15:42 (1763345742) [ 448.014031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.252177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.319520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 454.417542] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:15:49 (1763345749) [ 454.496534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.547995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.626402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 454.652613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 454.670945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 454.692456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 454.716795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 454.737146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 454.758686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 454.784790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 454.816952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 454.842435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 454.876864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 454.909615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 454.937830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 454.989192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 455.020953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 455.049947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 455.084894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 455.120271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 455.146633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 455.169083] LNetError: 28809:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 460.013409] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:15:55 (1763345755) [ 460.120538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.186036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.256051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.647494] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:16:00 (1763345760) [ 467.257540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg129-server [ 468.615655] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:16:03 (1763345763) [ 468.752551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.795126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.843566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.868160] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 470.271324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[30-34]@tcp [ 470.290738] LNet: 30524:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 470.350566] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 470.368208] LNetError: 30314:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.30@tcp has gone from up to down [ 471.336427] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 471.343936] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 472.354520] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 472.371507] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 474.413827] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.32@tcp is being used as a gateway but routing feature is not turned on [ 474.424592] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 476.156830] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:16:10 (1763345770) [ 477.647556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg129-server [ 478.500974] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.34@tcp is being used as a gateway but routing feature is not turned on [ 478.510810] LNetError: 30314:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 479.136820] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:16:13 (1763345773) [ 479.315977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.346522] LNetError: 31459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 479.372778] LNet: Removed LNI 192.168.201.29@tcp [ 479.398446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.437448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 479.465298] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 479.972825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[30-35/2]@tcp [ 479.999551] LNet: 31576:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 480.010423] LNet: 31576:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 480.028291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[30-35/2]@tcp [ 480.030602] LNetError: 31463:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.30@tcp has gone from up to down [ 480.051355] LNetError: 31463:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 485.579959] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:16:20 (1763345780) [ 485.729567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.772538] LNetError: 32164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 486.825347] LNet: Removed LNI 192.168.201.29@tcp [ 486.855915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.969871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 487.037155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.081131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.161503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.230159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.268883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.351890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 487.428547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.486429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.556453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.642621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.674980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.759591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 487.836692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.866453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.949803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.056913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.099601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.150845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 488.237960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.277229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.328895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.394859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.434133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.503968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 488.582608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.616686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.672018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.736745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.775643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.846227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 488.918598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.946699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.991811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.288744] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:16:29 (1763345789) [ 496.086804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg129-server [ 497.688093] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:16:32 (1763345792) [ 499.146710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg129-server [ 500.689115] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:16:35 (1763345795) [ 501.482940] Key type .llcrypt unregistered [ 501.484685] Key type ._llcrypt unregistered [ 505.861339] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 505.967874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 505.979878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 505.983138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 508.441470] Key type ._llcrypt registered [ 508.442879] Key type .llcrypt registered [ 508.616679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 508.637146] alg: No test for adler32 (adler32-zlib) [ 509.525520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.582863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 509.598838] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 510.179325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 510.194126] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 511.913808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 511.942399] LNet: Removed LNI 10.1.2.3@tcp [ 512.091718] LNet: 34286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.107427] LNetError: 34286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.120990] LNet: Removed LNI 192.168.201.29@tcp [ 512.714536] Key type .llcrypt unregistered [ 512.717051] Key type ._llcrypt unregistered [ 521.253933] Key type ._llcrypt registered [ 521.255165] Key type .llcrypt registered [ 522.550852] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:16:57 (1763345817) [ 522.965361] Key type .llcrypt unregistered [ 522.966857] Key type ._llcrypt unregistered [ 527.142101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 527.406793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 527.441901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 527.450742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 529.953526] Key type ._llcrypt registered [ 529.955013] Key type .llcrypt registered [ 530.200880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.236196] alg: No test for adler32 (adler32-zlib) [ 531.170736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.249227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 531.294807] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 531.851185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 531.863139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 534.273929] LNet: 35769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.279358] LNetError: 35769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.282580] LNet: Removed LNI 192.168.201.29@tcp [ 534.291580] LNet: Removed LNI 10.1.2.3@tcp [ 534.870365] Key type .llcrypt unregistered [ 534.871825] Key type ._llcrypt unregistered [ 544.770236] Key type ._llcrypt registered [ 544.772548] Key type .llcrypt registered [ 546.077912] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:17:20 (1763345840) [ 546.467424] Key type .llcrypt unregistered [ 546.470944] Key type ._llcrypt unregistered [ 550.496114] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 550.596270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 550.598240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 550.607158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 553.062238] Key type ._llcrypt registered [ 553.064540] Key type .llcrypt registered [ 553.272955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 553.289995] alg: No test for adler32 (adler32-zlib) [ 554.175161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.225710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 554.249926] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 555.723364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 555.739374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 557.411585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 557.434883] LNet: Removed LNI 10.1.2.3@tcp [ 557.457578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 557.487319] LNet: Removed LNI 10.1.2.31@tcp [ 557.502472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 557.528453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 558.086506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 558.102530] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 558.937061] LNet: 37679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.941496] LNetError: 37679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.945716] LNet: Removed LNI 10.1.2.3@tcp [ 558.960333] LNet: Removed LNI 10.1.2.31@tcp [ 559.432976] Key type .llcrypt unregistered [ 559.435898] Key type ._llcrypt unregistered [ 567.480636] Key type ._llcrypt registered [ 567.482209] Key type .llcrypt registered [ 568.733994] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:17:43 (1763345863) [ 568.997960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 569.016931] alg: No test for adler32 (adler32-zlib) [ 569.954497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.018555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.042335] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 570.765385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.788773] LNet: 38628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.792471] LNetError: 38628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.802538] LNet: Removed LNI 192.168.201.29@tcp [ 570.822660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.874785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.29@tcp [ 570.896179] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 571.570412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.29@tcp [ 571.592664] LNet: Removed LNI 192.168.201.29@tcp [ 571.661752] LNet: 38763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.222625] Key type .llcrypt unregistered [ 572.224941] Key type ._llcrypt unregistered [ 580.267765] Key type ._llcrypt registered [ 580.269441] Key type .llcrypt registered [ 581.119164] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:17:56 (1763345876) [ 582.285432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg129-server [ 583.561441] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:17:58 (1763345878) [ 583.883623] Key type .llcrypt unregistered [ 583.886380] Key type ._llcrypt unregistered [ 587.316322] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 587.485538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 587.488371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 587.491554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 589.905864] Key type ._llcrypt registered [ 589.907481] Key type .llcrypt registered [ 590.058554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 590.078664] alg: No test for adler32 (adler32-zlib) [ 590.971582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.005895] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 591.584294] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 593.198331] LNet: 40590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.203837] LNetError: 40590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.207352] LNet: Removed LNI 192.168.201.29@tcp1 [ 593.220995] LNet: Removed LNI 10.1.2.3@tcp2 [ 593.736433] Key type .llcrypt unregistered [ 593.738232] Key type ._llcrypt unregistered [ 602.120140] Key type ._llcrypt registered [ 602.121426] Key type .llcrypt registered [ 603.178192] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:18:18 (1763345898) [ 603.479745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.508315] alg: No test for adler32 (adler32-zlib) [ 604.426416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.453292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.468383] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 605.171702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.195728] LNet: 41555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.200989] LNetError: 41555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.214889] LNet: Removed LNI 192.168.201.29@tcp [ 605.238289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.307942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 605.336675] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 610.130982] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:18:25 (1763345905) [ 610.375043] LNet: 42146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.381807] LNetError: 42146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.399956] LNet: Removed LNI 192.168.201.29@tcp [ 610.867392] Key type .llcrypt unregistered [ 610.869774] Key type ._llcrypt unregistered [ 615.038945] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 615.219071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 615.221161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 615.223874] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 617.630149] Key type ._llcrypt registered [ 617.631803] Key type .llcrypt registered [ 617.792467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.803166] alg: No test for adler32 (adler32-zlib) [ 618.673260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.719667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.736716] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 619.368548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 619.387777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 621.480441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.533425] LNet: 42881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.548736] LNetError: 42881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.572478] LNet: Removed LNI 192.168.201.29@tcp [ 621.588336] LNet: Removed LNI 10.1.2.3@tcp [ 621.610101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.667492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 621.709455] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 621.715732] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 622.152908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.165812] LNet: 42908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.171312] LNetError: 42908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.174842] LNetError: 42908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 622.179777] LNet: Removed LNI 192.168.201.29@tcp [ 622.188505] LNet: Removed LNI 10.1.2.3@tcp [ 622.203761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.245185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 622.266814] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 622.273335] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 622.663289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 622.682670] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 623.740364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.759202] LNet: 43070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.769800] LNetError: 43070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.777138] LNetError: 43070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 623.788153] LNet: Removed LNI 192.168.201.29@tcp [ 623.790288] LNet: Removed LNI 10.1.2.3@tcp [ 623.808922] LNet: Removed LNI 192.168.201.29@tcp2 [ 623.828259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.868888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 623.893277] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 623.899218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 623.904935] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 629.619602] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:18:44 (1763345924) [ 629.732502] LNet: 43660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.739425] LNetError: 43660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.748605] LNetError: 43660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 629.754110] LNet: Removed LNI 192.168.201.29@tcp [ 629.756745] LNet: Removed LNI 10.1.2.3@tcp [ 629.777888] LNet: Removed LNI 192.168.201.29@tcp2 [ 630.377993] Key type .llcrypt unregistered [ 630.379553] Key type ._llcrypt unregistered [ 635.016229] Key type ._llcrypt registered [ 635.017931] Key type .llcrypt registered [ 635.263478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 635.276631] alg: No test for adler32 (adler32-zlib) [ 636.201946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.242189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 636.266559] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 637.135305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.171045] LNet: 44056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.177723] LNetError: 44056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.197595] LNet: Removed LNI 192.168.201.29@tcp [ 637.248704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.304814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 637.349899] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 637.858852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.887030] LNet: 44086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.895177] LNetError: 44086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.912801] LNet: Removed LNI 192.168.201.29@tcp [ 637.949599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.013613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 638.047121] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 638.676722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.727520] LNet: Removed LNI 192.168.201.29@tcp [ 638.744972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.825798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 638.848611] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 639.301258] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 639.428413] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 639.431077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 639.434379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 641.843891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.867836] LNet: 44195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.871362] LNet: 44195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 641.878774] LNetError: 44195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.883162] LNetError: 44195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 641.898955] LNet: Removed LNI 192.168.201.29@tcp [ 641.916290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.980714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 642.017855] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 648.328131] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:19:03 (1763345943) [ 648.494737] LNet: 44789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.506732] LNetError: 44789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.539852] LNet: Removed LNI 192.168.201.29@tcp [ 649.115779] Key type .llcrypt unregistered [ 649.120285] Key type ._llcrypt unregistered [ 654.342537] Key type ._llcrypt registered [ 654.349799] Key type .llcrypt registered [ 654.565802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.580602] alg: No test for adler32 (adler32-zlib) [ 655.498411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.558508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.590414] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 656.458554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.478546] LNet: 45186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.483566] LNetError: 45186:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.504370] LNet: Removed LNI 192.168.201.29@tcp [ 656.546217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.603224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 656.626731] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 662.371846] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:19:17 (1763345957) [ 662.464065] LNet: 45772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.472428] LNetError: 45772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.489764] LNet: Removed LNI 192.168.201.29@tcp [ 662.942508] Key type .llcrypt unregistered [ 662.944246] Key type ._llcrypt unregistered [ 666.790548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.864437] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.879319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.884977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.381940] Key type ._llcrypt registered [ 669.383489] Key type .llcrypt registered [ 669.482933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.495588] alg: No test for adler32 (adler32-zlib) [ 670.373349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.424559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.451399] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 671.335928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.380061] LNet: 46221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.389200] LNetError: 46221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.405560] LNet: Removed LNI 192.168.201.29@tcp [ 671.435948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.516171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 671.562875] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 677.987806] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:19:32 (1763345972) [ 678.146433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.184139] LNet: 46820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.198719] LNetError: 46820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.215059] LNet: Removed LNI 192.168.201.29@tcp [ 678.306738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.407672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 684.210912] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:19:39 (1763345979) [ 684.414224] LNet: 47397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.115599] Key type .llcrypt unregistered [ 685.117448] Key type ._llcrypt unregistered [ 691.063760] Key type ._llcrypt registered [ 691.065438] Key type .llcrypt registered [ 691.357669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 691.367524] alg: No test for adler32 (adler32-zlib) [ 692.335785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.381492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 692.405820] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 693.154020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.188162] LNet: 47795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.198249] LNetError: 47795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.220711] LNet: Removed LNI 192.168.201.29@tcp [ 693.238560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.303527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 693.332361] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 699.307665] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:19:54 (1763345994) [ 699.402888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.441308] LNet: 48382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.445868] LNetError: 48382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.477066] LNet: Removed LNI 192.168.201.29@tcp [ 699.512544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.585640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 705.198749] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:19:59 (1763345999) [ 706.669104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 708.583652] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:20:03 (1763346003) [ 710.095966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 711.546089] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:20:06 (1763346006) [ 712.990241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 714.467490] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:20:09 (1763346009) [ 715.779404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 717.357784] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:20:12 (1763346012) [ 718.932182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 720.730409] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:20:15 (1763346015) [ 720.831078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.845301] LNet: 50435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.864461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.928200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.964471] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 721.847570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.29 [ 721.876177] LNet: Removed LNI 192.168.201.29@tcp [ 727.478877] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:20:22 (1763346022) [ 727.761441] LNet: 51145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.770492] LNetError: 51145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.800986] LNet: Removed LNI 192.168.201.29@tcp [ 728.439731] Key type .llcrypt unregistered [ 728.444786] Key type ._llcrypt unregistered [ 733.619972] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 733.939171] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 733.941386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 733.976066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 736.764193] Key type ._llcrypt registered [ 736.765757] Key type .llcrypt registered [ 736.996605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.011084] alg: No test for adler32 (adler32-zlib) [ 738.431716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.440507] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:20:40 (1763346040) [ 746.689443] LNet: 52044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.696907] LNetError: 52044:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.710234] LNet: Removed LNI 10.1.2.3@tcp [ 747.334220] Key type .llcrypt unregistered [ 747.337613] Key type ._llcrypt unregistered [ 754.413628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.653529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.655460] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.657581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.352231] Key type ._llcrypt registered [ 757.353563] Key type .llcrypt registered [ 757.591790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.617691] alg: No test for adler32 (adler32-zlib) [ 758.572524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.843371] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:20:58 (1763346058) [ 764.014834] LNet: 52930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.026117] LNetError: 52930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 764.045133] LNet: Removed LNI 10.1.2.3@tcp [ 764.651385] Key type .llcrypt unregistered [ 764.655383] Key type ._llcrypt unregistered [ 769.984448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 770.264814] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 770.285623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 770.290926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.843563] Key type ._llcrypt registered [ 772.847503] Key type .llcrypt registered [ 773.013603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.032471] alg: No test for adler32 (adler32-zlib) [ 773.979783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 779.870452] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:21:14 (1763346074) [ 779.999826] LNet: 53816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.010928] LNetError: 53816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 780.048764] LNet: Removed LNI 10.1.2.3@tcp [ 780.610897] Key type .llcrypt unregistered [ 780.613483] Key type ._llcrypt unregistered [ 785.726681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.949857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.953720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.964678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.697696] Key type ._llcrypt registered [ 788.699394] Key type .llcrypt registered [ 788.971926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.982206] alg: No test for adler32 (adler32-zlib) [ 789.909674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.974224] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.355449] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:21:32 (1763346092) [ 797.783377] LNet: 55048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.791989] LNetError: 55048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 797.815782] LNet: Removed LNI 10.1.2.3@tcp [ 798.435594] Key type .llcrypt unregistered [ 798.437278] Key type ._llcrypt unregistered [ 804.233900] Key type ._llcrypt registered [ 804.235609] Key type .llcrypt registered [ 804.479972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.489979] alg: No test for adler32 (adler32-zlib) [ 805.424977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 805.446513] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 812.026180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 812.695307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_interrupt -r 1 [ 812.714958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_interrupt -r 1 [ 812.745386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 812.765908] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 812.959269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 812.981961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 813.003524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 813.035806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 813.065674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 813.087926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 813.130342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 813.168759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 813.185430] LNet: 55481:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 813.197671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 813.224947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 813.254519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 813.281793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 813.314945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 813.337623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 813.360672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 813.844950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_dropped -r 1 [ 813.888546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_dropped -r 1 [ 813.927764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 813.954801] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 814.218752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 814.240912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 814.289861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 814.318561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 814.338761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 814.382294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 814.409343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 814.442351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 814.456786] LNet: 55590:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 814.473176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 814.496230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 814.527440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 814.577255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 814.596624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 814.617782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 814.653507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 815.172350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_aborted -r 1 [ 815.223576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_aborted -r 1 [ 815.312880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 815.364841] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 815.697737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 815.712364] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 815.743743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 815.783146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 815.807425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 815.836656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 815.881518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 815.918256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 815.987431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 816.017597] LNet: 55700:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 816.081853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 816.177568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 816.266776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 816.315540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 816.366686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 816.435655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 816.475987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 817.119555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_no_route -r 1 [ 817.195408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_no_route -r 1 [ 817.255952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 817.678763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 817.738083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 817.785437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 817.808807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 817.841561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 817.906091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 817.943164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 817.985912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 818.040772] LNet: 55809:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 818.061016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 818.113755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 818.164334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 818.226887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 818.257908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 818.332307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 818.374907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 818.837853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_timeout -r 1 [ 818.874717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_timeout -r 1 [ 818.900628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 818.937658] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 818.940261] LNet: Skipped 1 previous similar message [ 819.143086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 819.186836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 819.232301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 819.281457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 819.317080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 819.337956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 819.364893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 819.412037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 819.458913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 819.501331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 819.521489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 819.540489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 819.562386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 819.594386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 819.616257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 820.158782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_error -r 1 [ 820.216230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_error -r 1 [ 820.250039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 820.544139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 820.574726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 820.635247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 820.685536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 820.730467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 820.788082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 820.839461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 820.879142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 820.921517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 820.951111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 820.978301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 821.019605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 821.068841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 821.119680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 821.140915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 821.427768] LNet: 56068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.440269] LNetError: 56068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.460529] LNet: Removed LNI 192.168.201.29@tcp [ 821.842839] Key type .llcrypt unregistered [ 821.844217] Key type ._llcrypt unregistered [ 841.312425] Key type ._llcrypt registered [ 841.316141] Key type .llcrypt registered [ 842.589858] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:22:17 (1763346137) [ 843.433313] Key type .llcrypt unregistered [ 843.434983] Key type ._llcrypt unregistered [ 848.658631] Key type ._llcrypt registered [ 848.660206] Key type .llcrypt registered [ 848.941111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.956759] alg: No test for adler32 (adler32-zlib) [ 849.831730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 849.888116] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 856.793360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 867.137671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.226261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 867.243723] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 868.900945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_interrupt -r 1 [ 868.973445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_interrupt -r 1 [ 869.048989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_interrupt -r 1 [ 869.106855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_interrupt -r 1 [ 869.146885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_interrupt -r 1 [ 869.200467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_interrupt -r 1 [ 869.250826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_interrupt -r 1 [ 869.275812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_interrupt -r 1 [ 869.314574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 869.328834] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 869.332693] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 869.571654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 869.590877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 869.615885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 869.644601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 869.661237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 869.691205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 869.719611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 869.745399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 869.756685] LNet: 57529:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 869.774161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 869.811953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 869.833614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 869.864852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 869.908218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 869.954188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 869.978696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 870.542439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_dropped -r 1 [ 870.583919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_dropped -r 1 [ 870.635616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_dropped -r 1 [ 870.689886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_dropped -r 1 [ 870.718490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_dropped -r 1 [ 870.805124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_dropped -r 1 [ 870.877135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_dropped -r 1 [ 870.940875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_dropped -r 1 [ 870.981693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 870.996757] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 871.000908] LNet: Skipped 2 previous similar messages [ 871.332524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 871.368562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 871.400486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 871.429262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 871.452628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 871.502878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 871.552366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 871.595560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 871.609670] LNet: 57650:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 871.623487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 871.667093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 871.697762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 871.732226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 871.755506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 871.798631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 871.843781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 872.375069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_aborted -r 1 [ 872.437519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_aborted -r 1 [ 872.494445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_aborted -r 1 [ 872.559953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_aborted -r 1 [ 872.634949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_aborted -r 1 [ 872.712133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_aborted -r 1 [ 872.806078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_aborted -r 1 [ 872.839151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_aborted -r 1 [ 872.904525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 872.934505] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 872.944467] LNet: Skipped 2 previous similar messages [ 873.219664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 873.264637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 873.315711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 873.355347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 873.401469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 873.430670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 873.472417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 873.506564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 873.538722] LNet: 57772:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.565658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 873.593193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 873.614892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 873.642102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 873.678467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 873.718698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 873.778847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 874.496444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_no_route -r 1 [ 874.531161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_no_route -r 1 [ 874.608239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_no_route -r 1 [ 874.663717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_no_route -r 1 [ 874.721227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_no_route -r 1 [ 874.789432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_no_route -r 1 [ 874.840899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_no_route -r 1 [ 874.888343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_no_route -r 1 [ 874.989584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 875.049260] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 875.072266] LNet: Skipped 2 previous similar messages [ 875.369620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 875.414201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 875.460768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 875.483354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 875.527061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 875.578097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 875.629191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 875.659559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 875.706948] LNet: 57893:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 875.756695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 875.833596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 875.903557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 875.968847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.033253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.061788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.094891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 876.662371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_timeout -r 1 [ 876.727523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_timeout -r 1 [ 876.780683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_timeout -r 1 [ 876.863250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_timeout -r 1 [ 876.938492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_timeout -r 1 [ 877.033788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_timeout -r 1 [ 877.099120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_timeout -r 1 [ 877.142128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_timeout -r 1 [ 877.205035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 877.460385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 877.481298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 877.509314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 877.549376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 877.582204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 877.628996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 877.725726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 877.797829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 877.857632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 877.905518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 877.956586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 878.036297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 878.077784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 878.116281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 878.156560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 878.752929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_error -r 1 [ 878.809632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_error -r 1 [ 878.837830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_error -r 1 [ 878.865256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_error -r 1 [ 878.898807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_error -r 1 [ 878.929911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_error -r 1 [ 878.994357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_error -r 1 [ 879.035321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_error -r 1 [ 879.076044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 879.086812] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 879.092690] LNet: Skipped 5 previous similar messages [ 879.310448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.362656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 879.405646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.445104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.490995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.525238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.571394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.607755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.642276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 879.674605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 879.729815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 879.788263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 879.836963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 879.874500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 879.907761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 885.323521] LNet: 58211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.330118] LNetError: 58211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.347206] LNet: Removed LNI 192.168.201.29@tcp [ 885.366879] LNet: Removed LNI 192.168.201.29@tcp1 [ 885.831681] Key type .llcrypt unregistered [ 885.837272] Key type ._llcrypt unregistered [ 901.167958] Key type ._llcrypt registered [ 901.170175] Key type .llcrypt registered [ 902.346711] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:23:17 (1763346197) [ 903.162646] Key type .llcrypt unregistered [ 903.170865] Key type ._llcrypt unregistered [ 908.375365] Key type ._llcrypt registered [ 908.377063] Key type .llcrypt registered [ 908.569701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.601483] alg: No test for adler32 (adler32-zlib) [ 909.700054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 909.738579] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 917.407200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 918.298635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_dropped -r 1 [ 918.337268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_dropped -r 1 [ 918.373816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 918.391861] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 918.632841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 918.690470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 918.754516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 918.817111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 918.852691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 918.908775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 918.965761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 919.000482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 919.029632] LNet: 59456:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 919.045275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 919.081665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 919.110535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 919.149097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 919.199449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.232639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 919.286815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 919.869837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_error -r 1 [ 919.929577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_error -r 1 [ 920.005894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 920.031981] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 920.247855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 920.274491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 920.301564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 920.326353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 920.364420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 920.443857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 920.488238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 920.514950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 920.547671] LNet: 59565:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 920.563100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 920.595384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 920.631090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 920.685386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 920.778714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 920.828582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 920.897589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 921.465389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_timeout -r 1 [ 921.498831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_timeout -r 1 [ 921.524458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 921.543413] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 921.809551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 921.854662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 921.883385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 921.923469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 921.951582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 921.976321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 922.058601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 922.115869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 922.159206] LNet: 59674:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 922.183595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 922.213588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 922.241354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 922.259375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 922.304371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 922.366589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 922.428324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 922.710813] LNet: 59714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.719389] LNetError: 59714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.733202] LNet: Removed LNI 192.168.201.29@tcp [ 923.185765] Key type .llcrypt unregistered [ 923.190409] Key type ._llcrypt unregistered [ 939.366083] Key type ._llcrypt registered [ 939.367805] Key type .llcrypt registered [ 940.610345] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:23:55 (1763346235) [ 941.418060] Key type .llcrypt unregistered [ 941.419487] Key type ._llcrypt unregistered [ 946.787201] Key type ._llcrypt registered [ 946.793045] Key type .llcrypt registered [ 946.965460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.981294] alg: No test for adler32 (adler32-zlib) [ 947.949497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 947.988114] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 954.561600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 962.274551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.307338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 962.318393] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 963.635966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_dropped -r 1 [ 963.681822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e remote_dropped -r 1 [ 963.722990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_dropped -r 1 [ 963.752892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e remote_dropped -r 1 [ 963.783612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e remote_dropped -r 1 [ 963.817630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e remote_dropped -r 1 [ 963.850817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e remote_dropped -r 1 [ 963.886288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e remote_dropped -r 1 [ 963.932860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 963.947300] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 963.965213] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.129@tcp [ 964.239712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 964.267842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 964.300266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 964.351023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 964.396013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 964.427487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 964.450220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 964.498704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 964.518605] LNet: 61177:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 964.540126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 964.566748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 964.596609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 964.624681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 964.645320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 964.676135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 964.711197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 964.751736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 965.178383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_error -r 1 [ 965.216903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e remote_error -r 1 [ 965.253397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_error -r 1 [ 965.280160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e remote_error -r 1 [ 965.311991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e remote_error -r 1 [ 965.340661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e remote_error -r 1 [ 965.373151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e remote_error -r 1 [ 965.402690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e remote_error -r 1 [ 965.440791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 965.462181] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 965.465494] LNet: Skipped 2 previous similar messages [ 965.685451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 965.728873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 965.775732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 965.797056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 965.813902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 965.841220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 965.869820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 965.896564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 965.908448] LNet: 61300:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 965.922381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 965.949544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 965.974988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 966.011086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 966.034565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 966.064245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 966.093403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 966.128490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 966.728577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_timeout -r 1 [ 966.758789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e remote_timeout -r 1 [ 966.797212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_timeout -r 1 [ 966.829285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e remote_timeout -r 1 [ 966.874659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e remote_timeout -r 1 [ 966.919318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e remote_timeout -r 1 [ 966.947169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e remote_timeout -r 1 [ 966.973641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e remote_timeout -r 1 [ 967.013362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.129@tcp [ 967.046052] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 967.055186] LNet: Skipped 1 previous similar message [ 967.261755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 967.296455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 967.330167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 967.355304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 967.377750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 967.404401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 967.430813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 967.469596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 967.496411] LNet: 61423:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.512909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 967.539179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 967.586950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 967.610239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 967.629352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 967.661727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 967.694478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 967.728055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.406886] LNet: 61500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.411508] LNetError: 61500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.415380] LNet: Removed LNI 192.168.201.29@tcp [ 971.427208] LNet: Removed LNI 192.168.201.29@tcp1 [ 971.722510] Key type .llcrypt unregistered [ 971.725401] Key type ._llcrypt unregistered [ 986.673756] Key type ._llcrypt registered [ 986.675194] Key type .llcrypt registered [ 988.104772] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:24:42 (1763346282) [ 988.600570] Key type .llcrypt unregistered [ 988.602156] Key type ._llcrypt unregistered [ 993.826370] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.990564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.994699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.000886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.604485] Key type ._llcrypt registered [ 996.610097] Key type .llcrypt registered [ 996.777459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.788955] alg: No test for adler32 (adler32-zlib) [ 997.680945] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 997.710993] LNet: 62589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.717633] LNetError: 62589:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.730681] LNet: Removed LNI 192.168.201.29@tcp [ 998.382621] Key type .llcrypt unregistered [ 998.387849] Key type ._llcrypt unregistered [ 1003.601988] Key type ._llcrypt registered [ 1003.603528] Key type .llcrypt registered [ 1003.776327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.792176] alg: No test for adler32 (adler32-zlib) [ 1004.713170] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1004.720230] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1004.759292] LNet: 62858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.765967] LNetError: 62858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.768520] LNet: Removed LNI 192.168.201.29@tcp [ 1004.785483] LNet: Removed LNI 10.1.2.3@tcp1 [ 1005.295406] Key type .llcrypt unregistered [ 1005.299086] Key type ._llcrypt unregistered [ 1011.310666] Key type ._llcrypt registered [ 1011.312355] Key type .llcrypt registered [ 1011.568251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.596472] alg: No test for adler32 (adler32-zlib) [ 1012.655612] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1012.667369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1012.713156] LNet: 63128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.718348] LNetError: 63128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.721944] LNet: Removed LNI 192.168.201.29@tcp [ 1012.731991] LNet: Removed LNI 10.1.2.3@tcp [ 1013.349875] Key type .llcrypt unregistered [ 1013.351684] Key type ._llcrypt unregistered [ 1020.858173] Key type ._llcrypt registered [ 1020.859626] Key type .llcrypt registered [ 1021.249136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.280558] alg: No test for adler32 (adler32-zlib) [ 1022.345458] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1022.347586] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1022.400285] LNet: 63399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.414587] LNetError: 63399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.423450] LNet: Removed LNI 192.168.201.29@tcp [ 1022.436459] LNet: Removed LNI 10.1.2.3@tcp [ 1023.160396] Key type .llcrypt unregistered [ 1023.163249] Key type ._llcrypt unregistered [ 1029.629714] Key type ._llcrypt registered [ 1029.631630] Key type .llcrypt registered [ 1029.844454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1029.868481] alg: No test for adler32 (adler32-zlib) [ 1030.789070] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1030.791888] LNetError: 63660:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1030.796831] LNetError: Error -100 starting up LNI tcp [ 1030.813875] LNet: Removed LNI 192.168.201.29@tcp [ 1031.478680] Key type .llcrypt unregistered [ 1031.482161] Key type ._llcrypt unregistered [ 1043.321692] Key type ._llcrypt registered [ 1043.327587] Key type .llcrypt registered [ 1044.930802] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:25:39 (1763346339) [ 1045.980499] Key type .llcrypt unregistered [ 1045.984733] Key type ._llcrypt unregistered [ 1052.274816] Key type ._llcrypt registered [ 1052.276774] Key type .llcrypt registered [ 1052.530076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.544670] alg: No test for adler32 (adler32-zlib) [ 1053.466475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1053.488950] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1060.259483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1060.945771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e network_timeout -r 1 [ 1060.997578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e network_timeout -r 1 [ 1061.055895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1061.084574] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1061.285582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.304246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1061.333907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.376074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.414042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.450223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.483304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.508588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.523478] LNet: 64910:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.543326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.578389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.614836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.649628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.670251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.709279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.727097] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1061.912927] LNet: 64948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.917362] LNetError: 64948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.925556] LNet: Removed LNI 192.168.201.29@tcp [ 1062.181720] Key type .llcrypt unregistered [ 1062.186600] Key type ._llcrypt unregistered [ 1075.573925] Key type ._llcrypt registered [ 1075.575402] Key type .llcrypt registered [ 1075.778890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.794839] alg: No test for adler32 (adler32-zlib) [ 1076.676430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1076.734663] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1083.277809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1090.631674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.681690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1090.713567] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1092.070968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e network_timeout -r 1 [ 1092.119167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e network_timeout -r 1 [ 1092.178753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e network_timeout -r 1 [ 1092.215497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e network_timeout -r 1 [ 1092.266396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e network_timeout -r 1 [ 1092.304550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e network_timeout -r 1 [ 1092.347849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e network_timeout -r 1 [ 1092.385240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e network_timeout -r 1 [ 1092.438800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1092.461440] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1092.722021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.773215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1092.793968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.855242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.916585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.959946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.994351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.025575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.046412] LNet: 65821:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.092623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.154697] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1093.157307] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.129@tcp [ 1093.162302] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1093.204571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.222115] LNet: Skipped 1 previous similar message [ 1093.224506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.282920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.331276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.382175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.438463] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1097.832460] LNet: 65894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.841600] LNetError: 65894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.853617] LNet: Removed LNI 192.168.201.29@tcp [ 1097.870172] LNet: Removed LNI 192.168.201.29@tcp1 [ 1098.373525] Key type .llcrypt unregistered [ 1098.375043] Key type ._llcrypt unregistered [ 1113.519470] Key type ._llcrypt registered [ 1113.522613] Key type .llcrypt registered [ 1114.751866] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:26:49 (1763346409) [ 1115.145124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.153643] alg: No test for adler32 (adler32-zlib) [ 1116.058451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.102032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1116.119256] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1116.770103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1116.790919] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1117.554208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1117.611451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1117.662938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1117.687947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1117.721456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1117.754725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1117.792856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.29@tcp [ 1117.807431] LNetError: 66992:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.201.29@tcp to recovery [ 1118.111364] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1118.114312] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1120.159464] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1121.252368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1124.257864] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1124.819525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1124.888586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.986875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.010337] LNet: 67099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.016322] LNetError: 67099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.021975] LNet: Removed LNI 192.168.201.29@tcp [ 1125.035810] LNet: Removed LNI 192.168.201.29@tcp1 [ 1125.062804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.121338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.164873] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1125.957326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1125.986899] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1126.782896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1126.834838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.935540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1126.984159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1127.022546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1127.060878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1127.113953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1127.169661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.29@tcp [ 1127.191391] LNetError: 67371:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.201.29@tcp to recovery [ 1128.163458] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1130.751547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1134.311178] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1134.314412] LNet: Skipped 1 previous similar message [ 1138.574479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1138.611254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.635212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.630138] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:27:19 (1763346439) [ 1144.718727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1144.738567] LNet: 68083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.744034] LNetError: 68083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.748046] LNetError: 68083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1144.751426] LNet: Removed LNI 192.168.201.29@tcp [ 1144.767470] LNet: Removed LNI 192.168.201.29@tcp1 [ 1144.783058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.837901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1144.864848] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1145.489230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1145.509599] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1146.375447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1146.435360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1146.471451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1146.497330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1146.554789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1146.601666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1146.667223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.29@tcp --health 0 [ 1146.792075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1146.847305] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.29@tcp [ 1147.872270] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1147.876054] LNet: Skipped 1 previous similar message [ 1163.143266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1163.259732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1163.285886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.310301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.29@tcp --health 500 [ 1163.396503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.908240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.978206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1169.996869] LNet: 68590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.004738] LNetError: 68590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.014935] LNetError: 68590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1170.020073] LNet: Removed LNI 192.168.201.29@tcp [ 1170.039489] LNet: Removed LNI 192.168.201.29@tcp1 [ 1170.059406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.112660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1170.147475] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1170.779573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1170.795932] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1171.547650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1171.586895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.632845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1171.654553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1171.683330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1171.757860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1171.796353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1171.843195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.29@tcp --health 0 [ 1172.128917] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1172.134433] LNet: Skipped 3 previous similar messages [ 1173.161701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1183.383578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1183.415982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.456642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.193241] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:28:04 (1763346484) [ 1189.743812] LNet: 69635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.750283] LNetError: 69635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.754209] LNetError: 69635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1189.760654] LNet: Removed LNI 192.168.201.29@tcp [ 1189.777914] LNet: Removed LNI 192.168.201.29@tcp1 [ 1190.320607] Key type .llcrypt unregistered [ 1190.322984] Key type ._llcrypt unregistered [ 1195.079258] Key type ._llcrypt registered [ 1195.081549] Key type .llcrypt registered [ 1195.249836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.267488] alg: No test for adler32 (adler32-zlib) [ 1196.166678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.195811] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1203.397431] Lustre: DEBUG MARKER: Initial discovery [ 1203.440801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.129@tcp [ 1205.558494] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1205.657443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1205.699756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.129@tcp [ 1205.727871] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1205.758189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1205.832157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.129@tcp -r 1 -m GET -l 3 [ 1205.878364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.129@tcp -r 1 -m GET -e local_error [ 1205.931699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.129@tcp -r 1 -m PUT -l 6 [ 1206.434556] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1207.173924] Lustre: DEBUG MARKER: Force oleg129-client.virtnet to discover 192.168.201.129@tcp (in background) [ 1207.291927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.129@tcp [ 1208.566616] Lustre: DEBUG MARKER: Force oleg129-server to discover 192.168.201.29@tcp [ 1210.337823] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1210.684087] Lustre: DEBUG MARKER: Wait for 70386 [ 1211.799198] Lustre: DEBUG MARKER: Finished wait on 70386 [ 1211.894192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1211.922870] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1211.925604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.185643] LNet: 70843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.197055] LNetError: 70843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.221747] LNet: Removed LNI 192.168.201.29@tcp [ 1213.623423] Key type .llcrypt unregistered [ 1213.625136] Key type ._llcrypt unregistered [ 1226.136747] Key type ._llcrypt registered [ 1226.138872] Key type .llcrypt registered [ 1227.228835] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:28:42 (1763346522) [ 1227.638644] Key type .llcrypt unregistered [ 1227.642712] Key type ._llcrypt unregistered [ 1231.879820] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1232.032769] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1232.038668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1232.041360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1234.499398] Key type ._llcrypt registered [ 1234.501458] Key type .llcrypt registered [ 1234.689096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.722688] alg: No test for adler32 (adler32-zlib) [ 1235.638653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.697088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1235.731790] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1236.428361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1236.449810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1243.440398] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:28:58 (1763346538) [ 1243.556648] LNet: 72905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.577871] LNetError: 72905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.588208] LNet: Removed LNI 192.168.201.29@tcp [ 1243.613465] LNet: Removed LNI 10.1.2.3@tcp [ 1244.268454] Key type .llcrypt unregistered [ 1244.274715] Key type ._llcrypt unregistered [ 1250.580608] Key type ._llcrypt registered [ 1250.582171] Key type .llcrypt registered [ 1250.869389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.918182] alg: No test for adler32 (adler32-zlib) [ 1251.932180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.981912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1252.005351] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1252.833842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1252.864328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1254.000037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.29@tcp [ 1255.071187] LNetError: Unexpected error -22 connecting to 192.168.201.29@tcp at host 192.168.201.29:988 [ 1260.092598] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:29:14 (1763346554) [ 1260.213972] LNet: 74031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.230602] LNetError: 74031:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.238969] LNet: Removed LNI 192.168.201.29@tcp [ 1260.253221] LNet: Removed LNI 10.1.2.3@tcp [ 1260.890476] Key type .llcrypt unregistered [ 1260.892134] Key type ._llcrypt unregistered [ 1265.924553] Key type ._llcrypt registered [ 1265.926037] Key type .llcrypt registered [ 1266.115898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.126456] alg: No test for adler32 (adler32-zlib) [ 1267.027559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.085514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.121564] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1267.698589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1267.719732] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1268.479776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.29@tcp1 --nid 192.168.201.29@tcp2 [ 1272.152876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp1 192.168.201.29@tcp2 [ 1272.196249] LNetError: 74872:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.29@tcp1 to 192.168.201.29@tcp2 - Route table may be misconfigured [ 1272.212435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp2 192.168.201.29@tcp1 [ 1277.472065] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:29:32 (1763346572) [ 1277.638565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.659860] LNet: 75443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.668689] LNetError: 75443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.687293] LNet: Removed LNI 192.168.201.29@tcp1 [ 1278.693106] LNetError: 75443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.705793] LNet: Removed LNI 192.168.201.29@tcp2 [ 1278.734174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.779758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.804476] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1279.500130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.512117] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1280.301730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1280.335683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.29@tcp [ 1280.381078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp1 192.168.201.29@tcp1 [ 1280.428547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.29@tcp -d 192.168.201.29@tcp -e network_timeout [ 1280.471624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -e network_timeout [ 1280.501625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -e network_timeout [ 1280.529538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -e network_timeout [ 1280.566846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1280.581726] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1280.597343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.674659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1280.762820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.800325] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1285.932819] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:29:40 (1763346580) [ 1286.072336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1286.100867] LNet: 76303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.121898] LNetError: 76303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.135170] LNet: Removed LNI 192.168.201.29@tcp [ 1288.168860] LNet: Removed LNI 192.168.201.29@tcp1 [ 1288.220302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.315308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1288.522877] LNet: 76342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.957425] Key type .llcrypt unregistered [ 1288.958808] Key type ._llcrypt unregistered [ 1299.207389] Key type ._llcrypt registered [ 1299.208796] Key type .llcrypt registered [ 1300.533958] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:29:55 (1763346595) [ 1300.950287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.975162] alg: No test for adler32 (adler32-zlib) [ 1301.972451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.865464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1305.584418] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:30:00 (1763346600) [ 1305.717194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.728703] LNet: 77436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.757558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.798742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.821982] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1306.504327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.519588] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1307.203148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1307.251990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp1 [ 1307.289920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp1 [ 1312.326549] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:30:07 (1763346607) [ 1313.997264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg129-server [ 1315.392549] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:30:10 (1763346610) [ 1317.136202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg129-server [ 1318.683936] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:30:13 (1763346613) [ 1320.867358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg129-server [ 1322.319627] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:30:17 (1763346617) [ 1323.909650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg129-server [ 1325.434963] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:30:20 (1763346620) [ 1326.823472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg129-server [ 1328.219666] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:30:23 (1763346623) [ 1329.561544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg129-server [ 1330.828582] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:30:25 (1763346625) [ 1332.186848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg129-server [ 1333.506374] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:30:28 (1763346628) [ 1335.287674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg129-server [ 1336.755944] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:30:31 (1763346631) [ 1338.129791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg129-server [ 1339.307862] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:30:34 (1763346634) [ 1339.443800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.464683] LNet: 81313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.475609] LNetError: 81313:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.482723] LNet: Removed LNI 192.168.201.29@tcp [ 1340.519173] LNet: Removed LNI 192.168.201.29@tcp1 [ 1340.536473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.576969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.598625] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1341.043282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1341.106599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1341.223740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1341.239933] LNet: 81460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.244299] LNetError: 81460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.249345] LNetError: 81460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1341.259771] LNet: Removed LNI 192.168.201.29@tcp [ 1341.283554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.330099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1341.342094] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1342.012407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1342.161139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.206111] LNet: Removed LNI 192.168.201.29@tcp [ 1343.223965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.269807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1343.289416] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1343.841586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1343.897563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1345.110318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.127121] LNet: 81762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.130627] LNet: 81762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1345.136955] LNetError: 81762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.140114] LNetError: 81762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1345.157454] LNet: Removed LNI 192.168.201.29@tcp [ 1345.173265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.216643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1345.232734] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1345.754331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1345.898311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.988061] LNet: Removed LNI 192.168.201.29@tcp [ 1347.022323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.093075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1347.123321] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1347.649414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1347.722946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1347.885270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.901724] LNetError: 82054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.908629] LNetError: 82054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1348.966246] LNet: Removed LNI 192.168.201.29@tcp [ 1348.982593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.020335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1349.043224] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1349.603351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1351.911781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.930119] LNet: 82218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.933079] LNet: 82218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1351.940098] LNetError: 82218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.952942] LNet: Removed LNI 192.168.201.29@tcp [ 1351.967128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.003230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1352.019991] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1352.451783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1352.504325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1352.672476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.705476] LNet: Removed LNI 192.168.201.29@tcp [ 1353.732615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.775864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1353.800482] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1354.272835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1354.447631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.496133] LNet: Removed LNI 192.168.201.29@tcp [ 1355.525242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.583274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1355.620415] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1356.352796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1356.401501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1356.564604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.605588] LNet: Removed LNI 192.168.201.29@tcp [ 1357.636954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.690241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1357.721099] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1358.325511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1358.560240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.596575] LNet: Removed LNI 192.168.201.29@tcp [ 1358.623102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.686831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1358.711995] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1359.331627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1359.423119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1359.746307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1360.810857] LNet: Removed LNI 192.168.201.29@tcp [ 1360.828153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.871808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1360.909366] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1361.674901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1362.008000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.047608] LNet: 83095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.061202] LNet: 83095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1362.076173] LNetError: 83095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.083601] LNetError: 83095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1362.109381] LNet: Removed LNI 192.168.201.29@tcp [ 1362.155244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.218137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.259214] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1363.026617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1363.093315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.128764] LNet: Removed LNI 192.168.201.29@tcp [ 1363.146599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.201709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.252116] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1363.986217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1369.209951] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:31:04 (1763346664) [ 1369.268322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.286816] LNet: Removed LNI 192.168.201.29@tcp [ 1369.302895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.342996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1369.358338] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/180] [ 1370.159043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.189669] LNet: Removed LNI 192.168.201.29@tcp231 [ 1370.209093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.268188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1370.296575] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/180] [ 1370.303524] LNet: 84078:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.596860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1370.624273] LNet: Removed LNI 192.168.201.29@tcp231 [ 1370.650461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1370.673498] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/180] [ 1371.465202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.498177] LNet: Removed LNI 192.168.201.29@tcp231 [ 1371.512940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.575417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1371.596096] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/0] [ 1371.599459] LNet: 84231:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.870153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1371.892991] LNet: Removed LNI 192.168.201.29@tcp231 [ 1371.907126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1371.923211] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/0] [ 1372.732607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.760792] LNet: Removed LNI 192.168.201.29@tcp231 [ 1372.784240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.833967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1372.854490] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/60] [ 1372.858546] LNet: 84382:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1373.182928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1373.207530] LNet: Removed LNI 192.168.201.29@tcp231 [ 1373.222536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1373.257376] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/60] [ 1374.044278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.062514] LNet: Removed LNI 192.168.201.29@tcp231 [ 1374.077788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.133465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1374.163206] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/180] [ 1374.437528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1374.467916] LNet: Removed LNI 192.168.201.29@tcp231 [ 1374.477994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1374.493428] LNet: Added LNI 192.168.201.29@tcp231 [8/256/0/180] [ 1379.443618] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:31:14 (1763346674) [ 1379.575609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.606328] LNet: 85243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.612384] LNet: 85243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1379.617203] LNetError: 85243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.619944] LNetError: 85243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1379.631599] LNet: Removed LNI 192.168.201.29@tcp231 [ 1379.649348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.704508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1379.720978] LNet: Added LNI 192.168.201.29@tcp232 [8/256/0/180] [ 1380.366182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1380.388739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1380.416775] LNet: Removed LNI 192.168.201.29@tcp232 [ 1380.435140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1380.460136] LNet: Added LNI 192.168.201.29@tcp232 [8/256/0/180] [ 1381.004096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1381.025412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1381.050335] LNet: Removed LNI 192.168.201.29@tcp232 [ 1381.071968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1381.101102] LNet: Added LNI 192.168.201.29@tcp232 [8/256/0/180] [ 1381.673346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1381.704710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1381.723693] LNet: Removed LNI 192.168.201.29@tcp232 [ 1381.735097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1381.753335] LNet: Added LNI 192.168.201.29@tcp232 [8/256/0/180] [ 1382.316891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1382.354960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1382.386165] LNet: Removed LNI 192.168.201.29@tcp232 [ 1382.406682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1387.381220] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:31:22 (1763346682) [ 1388.379526] Key type .llcrypt unregistered [ 1388.382908] Key type ._llcrypt unregistered [ 1392.662076] Key type ._llcrypt registered [ 1392.663680] Key type .llcrypt registered [ 1392.800608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1392.810385] alg: No test for adler32 (adler32-zlib) [ 1393.702950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1393.728474] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1399.391519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1405.305520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.356523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1405.373443] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1406.524537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_interrupt -r 2 [ 1406.562201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_interrupt -r 2 [ 1406.592578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_interrupt -r 2 [ 1406.628166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_interrupt -r 2 [ 1406.667501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_interrupt -r 2 [ 1406.694064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_interrupt -r 2 [ 1406.734744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_interrupt -r 2 [ 1406.765250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_interrupt -r 2 [ 1406.809880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1406.833905] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1406.837419] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1407.040269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1407.064062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1407.082239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1407.112475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1407.143355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1407.168550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1407.195911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1407.234336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1407.262611] LNet: 87027:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1407.278582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1407.308495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1407.337695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1407.389964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1407.417334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1407.452083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1407.807888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_dropped -r 2 [ 1407.835957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_dropped -r 2 [ 1407.858516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_dropped -r 2 [ 1407.884073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_dropped -r 2 [ 1407.923970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_dropped -r 2 [ 1407.951721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_dropped -r 2 [ 1407.992565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_dropped -r 2 [ 1408.023438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_dropped -r 2 [ 1408.060954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1408.079572] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1408.299284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1408.318810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1408.345826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.374714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.399773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.419653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.445210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.478800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.510475] LNet: 87147:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.528100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.565655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.586874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.614768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.640495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1408.663621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1409.109802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_aborted -r 2 [ 1409.142656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_aborted -r 2 [ 1409.171938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_aborted -r 2 [ 1409.208577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_aborted -r 2 [ 1409.236406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_aborted -r 2 [ 1409.268481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_aborted -r 2 [ 1409.291635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_aborted -r 2 [ 1409.319673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_aborted -r 2 [ 1409.342574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1409.354199] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1409.531731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1409.558437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1409.585585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1409.602640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1409.623861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1409.645864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1409.664449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1409.695522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1409.711174] LNet: 87267:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1409.728253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1409.759066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1409.790101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.819930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.851381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1409.889649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1410.329316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_no_route -r 2 [ 1410.358582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_no_route -r 2 [ 1410.385939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_no_route -r 2 [ 1410.418790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_no_route -r 2 [ 1410.458421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_no_route -r 2 [ 1410.484405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_no_route -r 2 [ 1410.517182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_no_route -r 2 [ 1410.537335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_no_route -r 2 [ 1410.575967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1410.789715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.826210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1410.857334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.885562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.925360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.956478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1411.004290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1411.034935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1411.067494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1411.104281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1411.129225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1411.153742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1411.196285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.247242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1411.611144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e local_timeout -r 2 [ 1411.638669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e local_timeout -r 2 [ 1411.668522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e local_timeout -r 2 [ 1411.702133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e local_timeout -r 2 [ 1411.742971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e local_timeout -r 2 [ 1411.773580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e local_timeout -r 2 [ 1411.809206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e local_timeout -r 2 [ 1411.843083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e local_timeout -r 2 [ 1411.873034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1411.889504] LNet: There was an unexpected network error while writing to 192.168.201.129: rc = -22 [ 1411.896048] LNet: Skipped 1 previous similar message [ 1412.059503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1412.084489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1412.102665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.124580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.154968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1412.177378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1412.205550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1412.245626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1412.263632] LNet: 87508:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1412.267985] LNet: 87508:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1412.281251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1412.303525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1412.346410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.380199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1412.414515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1412.440808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1412.902555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp -m GET -e remote_dropped -r 2 [ 1412.922484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.129@tcp1 -m GET -e remote_dropped -r 2 [ 1412.954434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp -m GET -e remote_dropped -r 2 [ 1412.978261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp -d 192.168.201.29@tcp1 -m GET -e remote_dropped -r 2 [ 1413.003582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp -m GET -e remote_dropped -r 2 [ 1413.028805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.129@tcp1 -m GET -e remote_dropped -r 2 [ 1413.059700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp -m GET -e remote_dropped -r 2 [ 1413.083592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.29@tcp1 -d 192.168.201.29@tcp1 -m GET -e remote_dropped -r 2 [ 1413.105592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.29@tcp 192.168.201.129@tcp [ 1413.124493] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.129@tcp [ 1413.271189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1413.290304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1413.311454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1413.334438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1413.354638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1413.372267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1413.402679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1413.422636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1413.452724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1413.479196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1413.502351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.522139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1413.545559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1413.574670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1416.995200] LNet: 87700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.005538] LNetError: 87700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.010552] LNet: Removed LNI 192.168.201.29@tcp [ 1417.026291] LNet: Removed LNI 192.168.201.29@tcp1 [ 1417.328400] Key type .llcrypt unregistered [ 1417.330279] Key type ._llcrypt unregistered [ 1428.021148] Key type ._llcrypt registered [ 1428.022692] Key type .llcrypt registered [ 1428.852143] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:32:04 (1763346724) [ 1429.074547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1429.091467] alg: No test for adler32 (adler32-zlib) [ 1429.952230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.998335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.025556] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1430.667517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1430.692536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.29@tcp [ 1430.711018] LNetError: 88675:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.201.29@tcp to recovery [ 1430.777685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.29@tcp [ 1430.866269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.951691] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:32:10 (1763346730) [ 1435.051755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.068539] LNet: 89261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.073182] LNetError: 89261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.089038] LNet: Removed LNI 192.168.201.29@tcp [ 1435.106373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.145623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.160372] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1435.677252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1435.691667] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1436.368785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.29@tcp [ 1436.382838] LNetError: 89530:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.201.29@tcp to recovery [ 1437.154262] LNet: 1 local NIs in recovery (showing 1): 192.168.201.29@tcp [ 1437.156894] LNet: There was an unexpected network error while writing to 192.168.201.29: rc = -22 [ 1437.555826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.29@tcp [ 1445.544716] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:32:20 (1763346740) [ 1447.294130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg129-server [ 1449.014706] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:32:23 (1763346743) [ 1449.102882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.125124] LNet: 90479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.131228] LNetError: 90479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.143288] LNet: Removed LNI 192.168.201.29@tcp [ 1450.145313] LNetError: 90479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.162460] LNet: Removed LNI 192.168.201.29@tcp1 [ 1450.181343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.258794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.283555] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1450.961509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1451.016048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1451.058410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1451.113657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1451.154270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1451.189212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1456.602456] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:32:31 (1763346751) [ 1456.694330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.709878] LNet: 91205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.714443] LNetError: 91205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.725677] LNet: Removed LNI 192.168.201.29@tcp [ 1456.759585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.851397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.873980] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1462.617837] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:32:37 (1763346757) [ 1462.726716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.756956] LNet: 91909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.761825] LNetError: 91909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.788100] LNet: Removed LNI 192.168.201.29@tcp [ 1462.816713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.886692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.909096] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1469.073942] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:32:43 (1763346763) [ 1470.433415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1471.903777] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:32:46 (1763346766) [ 1472.274482] LNet: 92948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.279276] LNetError: 92948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.289630] LNet: Removed LNI 192.168.201.29@tcp [ 1472.810801] Key type .llcrypt unregistered [ 1472.819647] Key type ._llcrypt unregistered [ 1477.719111] Key type ._llcrypt registered [ 1477.721266] Key type .llcrypt registered [ 1477.929626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1477.962159] alg: No test for adler32 (adler32-zlib) [ 1478.887410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1478.904713] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1484.664968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1487.733866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.129@tcp [ 1487.988959] LNet: 93359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.995525] LNetError: 93359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.010298] LNet: Removed LNI 192.168.201.29@tcp [ 1488.329744] Key type .llcrypt unregistered [ 1488.331435] Key type ._llcrypt unregistered [ 1496.299320] Key type ._llcrypt registered [ 1496.300975] Key type .llcrypt registered [ 1497.580437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1498.822033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1499.941849] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:33:15 (1763346795) [ 1500.329984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.346716] alg: No test for adler32 (adler32-zlib) [ 1501.241560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.335641] LNet: 94442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.855516] Key type .llcrypt unregistered [ 1501.857089] Key type ._llcrypt unregistered [ 1506.645916] Key type ._llcrypt registered [ 1506.648677] Key type .llcrypt registered [ 1506.892522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.915137] alg: No test for adler32 (adler32-zlib) [ 1507.828211] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1507.830767] LNet: 94694:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1507.855633] LNetError: 94706:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 1507.868850] LNetError: 94706:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.30@tcp has gone from up to down [ 1508.326291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.358928] LNet: 94752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.370638] LNetError: 94752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.392798] LNet: Removed LNI 192.168.201.29@tcp [ 1508.414856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.477211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1508.509990] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1508.516946] LNet: 94759:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1508.523187] LNet: 94759:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1508.533474] LNetError: 94759:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.31@tcp is a gateway. Can not delete it [ 1508.546451] LNetError: 94759:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.201.31@tcp is a router. Can not be deleted [ 1508.558486] LNet: 94759:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1508.561333] LNetError: 94756:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.33@tcp is being used as a gateway but routing feature is not turned on [ 1508.583880] LNetError: 94756:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1508.588353] LNetError: 94756:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.33@tcp has gone from up to down [ 1508.593454] LNetError: 94756:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1514.507903] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:33:29 (1763346809) [ 1516.319377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg129-server [ 1517.687585] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:33:32 (1763346812) [ 1519.238645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg129-server [ 1520.810701] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:33:35 (1763346815) [ 1522.349898] Lustre: Lustre: Build Version: 2.16.61_45_g58940e0 [ 1524.247180] Key type lgssc registered [ 1525.411590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.017474] Key type lgssc unregistered [ 1541.305189] LNet: 97177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.317136] LNetError: 97177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.338987] LNet: Removed LNI 192.168.201.29@tcp [ 1541.752853] Key type .llcrypt unregistered [ 1541.754643] Key type ._llcrypt unregistered [ 1552.192273] Key type ._llcrypt registered [ 1552.194686] Key type .llcrypt registered [ 1553.198722] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:34:08 (1763346848) [ 1553.561848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.575606] alg: No test for adler32 (adler32-zlib) [ 1554.463270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.540765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.597864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.671375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1554.724180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.668524] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:34:15 (1763346855) [ 1560.832587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.852865] LNet: 98581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.874242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.916974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.936657] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1561.728373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.749895] LNet: 98725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.754994] LNetError: 98725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.767054] LNet: Removed LNI 192.168.201.29@tcp [ 1561.802928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.842667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1561.863806] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1562.237398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.254194] LNetError: 98753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.266184] LNet: Removed LNI 192.168.201.29@tcp [ 1562.281282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.325956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1562.351720] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1567.551458] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:34:22 (1763346862) [ 1567.709356] LNet: 99336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.720133] LNet: 99336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1567.730806] LNetError: 99336:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.742360] LNet: Removed LNI 192.168.201.29@tcp [ 1568.281558] Key type .llcrypt unregistered [ 1568.287116] Key type ._llcrypt unregistered [ 1572.442337] Key type ._llcrypt registered [ 1572.445083] Key type .llcrypt registered [ 1572.621815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1572.640341] alg: No test for adler32 (adler32-zlib) [ 1573.580428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.662610] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1573.664748] LNet: *** cfs_fail_loc=e003, val=100*** [ 1574.287843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1574.309523] LNet: *** cfs_fail_loc=e003, val=100*** [ 1574.312186] LNet: Skipped 2 previous similar messages [ 1574.319661] Ping reply received with mapping: [ 1574.326794] found metadata reply NI[192.168.201.29@tcp] [ 1574.332473] 1d c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.338261] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.342424] 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.489628] LNet: 99738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.496776] LNetError: 99738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.521965] LNet: Removed LNI 192.168.201.29@tcp [ 1575.138418] Key type .llcrypt unregistered [ 1575.140117] Key type ._llcrypt unregistered [ 1583.911544] Key type ._llcrypt registered [ 1583.919423] Key type .llcrypt registered [ 1585.146610] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:34:39 (1763346879) [ 1585.522410] Key type .llcrypt unregistered [ 1585.524487] Key type ._llcrypt unregistered [ 1589.554061] Key type ._llcrypt registered [ 1589.559096] Key type .llcrypt registered [ 1589.727596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.748636] alg: No test for adler32 (adler32-zlib) [ 1590.830090] LNetError: 100773:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1591.188818] Key type .llcrypt unregistered [ 1591.190844] Key type ._llcrypt unregistered [ 1595.183761] Key type ._llcrypt registered [ 1595.185438] Key type .llcrypt registered [ 1596.082977] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:34:51 (1763346891) [ 1596.572659] Key type .llcrypt unregistered [ 1596.575479] Key type ._llcrypt unregistered [ 1600.571686] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1600.578218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1602.960487] Key type ._llcrypt registered [ 1602.962235] Key type .llcrypt registered [ 1603.090821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1603.102542] alg: No test for adler32 (adler32-zlib) [ 1603.982825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.026512] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1611.677977] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:35:06 (1763346906) [ 1611.756639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.789571] LNet: 102537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.814560] LNetError: 102537:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1611.845570] LNet: Removed LNI 10.1.2.3@tcp [ 1611.875541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.447588] Key type .llcrypt unregistered [ 1612.449274] Key type ._llcrypt unregistered [ 1616.921488] Key type ._llcrypt registered [ 1616.923110] Key type .llcrypt registered [ 1617.070823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1617.083126] alg: No test for adler32 (adler32-zlib) [ 1618.002202] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1618.046730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1618.169401] LNet: 102830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.173567] LNetError: 102830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.189891] LNet: Removed LNI 192.168.201.29@tcp [ 1618.645095] Key type .llcrypt unregistered [ 1618.650237] Key type ._llcrypt unregistered [ 1628.896684] Key type ._llcrypt registered [ 1628.898756] Key type .llcrypt registered [ 1630.189055] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:35:24 (1763346924) [ 1630.831477] Key type .llcrypt unregistered [ 1630.835514] Key type ._llcrypt unregistered [ 1635.364197] Key type ._llcrypt registered [ 1635.365758] Key type .llcrypt registered [ 1635.560251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1635.579647] alg: No test for adler32 (adler32-zlib) [ 1641.957539] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:35:36 (1763346936) [ 1642.125590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.160622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.233941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.272190] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1642.940690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.984503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1643.023143] LNet: Removed LNI 192.168.201.29@tcp [ 1643.237662] LNet: 104566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.541322] Key type .llcrypt unregistered [ 1643.542646] Key type ._llcrypt unregistered [ 1653.150382] Key type ._llcrypt registered [ 1653.153634] Key type .llcrypt registered [ 1654.191491] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:35:49 (1763346949) [ 1654.699964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1654.717122] alg: No test for adler32 (adler32-zlib) [ 1655.647538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.691320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.710171] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1656.304990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1656.336879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.29@tcp [ 1656.352828] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1656.356180] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1656.358412] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1656, 0, 0 [ 1656.361977] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1656.365338] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1656.369299] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1656.375965] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1656, 0, 0 [ 1656.379356] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1656.384338] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1656.387813] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1656.390645] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1656, 0, 1706 [ 1656.395076] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1656.397916] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1656.401761] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1656.411439] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1656, 0, 1706 [ 1656.414576] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1661.015292] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:35:56 (1763346956) [ 1661.449687] LNet: 106102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.457928] LNetError: 106102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.469245] LNet: Removed LNI 192.168.201.29@tcp [ 1661.918356] Key type .llcrypt unregistered [ 1661.923331] Key type ._llcrypt unregistered [ 1666.023528] Key type ._llcrypt registered [ 1666.025062] Key type .llcrypt registered [ 1666.191518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.201621] alg: No test for adler32 (adler32-zlib) [ 1667.075458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1667.099171] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1672.883246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1679.655426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.704719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1679.727186] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1680.441493] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1680.564368] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1680.566778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1680.569404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1682.838832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1682.859919] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1690.255484] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:36:25 (1763346985) [ 1690.357277] LNet: 107596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.365953] LNetError: 107596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.373701] LNet: Removed LNI 192.168.201.29@tcp [ 1691.424079] LNet: Removed LNI 192.168.201.29@tcp1 [ 1691.426136] LNetError: 107596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1691.431691] LNetError: 107596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.444556] LNet: Removed LNI 10.1.2.3@tcp99 [ 1691.920780] Key type .llcrypt unregistered [ 1691.922277] Key type ._llcrypt unregistered [ 1695.909076] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1695.912317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1698.275971] Key type ._llcrypt registered [ 1698.277557] Key type .llcrypt registered [ 1698.431496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1698.443422] alg: No test for adler32 (adler32-zlib) [ 1699.319881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.370615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.404464] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1700.048823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1700.060481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1701.581320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1701.618667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1701.665529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1701.681499] LNetError: 108341:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1701.712801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1701.745487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1702.054906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1702.105930] LNet: 108361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.110454] LNetError: 108361:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.116535] LNet: Removed LNI 192.168.201.29@tcp [ 1702.136443] LNet: Removed LNI 10.1.2.3@tcp [ 1702.637249] Key type .llcrypt unregistered [ 1702.642456] Key type ._llcrypt unregistered [ 1710.584391] Key type ._llcrypt registered [ 1710.585958] Key type .llcrypt registered [ 1711.683972] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:36:46 (1763347006) [ 1711.906635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.921290] alg: No test for adler32 (adler32-zlib) [ 1712.826403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.866246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.890261] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1713.441467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.29@tcp [ 1713.503756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg129-client@tcp [ 1718.256987] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:36:53 (1763347013) [ 1718.364775] LNet: 109885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.375351] LNetError: 109885:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.403352] LNet: Removed LNI 192.168.201.29@tcp [ 1719.890396] Key type .llcrypt unregistered [ 1719.892051] Key type ._llcrypt unregistered [ 1724.556284] Key type ._llcrypt registered [ 1724.566240] Key type .llcrypt registered [ 1724.692959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.702871] alg: No test for adler32 (adler32-zlib) [ 1725.658248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1725.702110] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1725.732850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1731.106839] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:37:06 (1763347026) [ 1732.298911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1733.535384] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:37:08 (1763347028) [ 1734.944418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1736.432410] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:37:11 (1763347031) [ 1736.491962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.510477] LNet: 111299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.526280] LNetError: 111299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.545023] LNet: Removed LNI 192.168.201.29@tcp [ 1736.586730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.673637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.698830] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1737.443766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1737.487986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1743.437684] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:37:18 (1763347038) [ 1743.714771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.739571] LNet: 112016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.747138] LNetError: 112016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.764510] LNet: Removed LNI 192.168.201.29@tcp [ 1743.779737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.829035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1743.862817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1744.634368] Key type .llcrypt unregistered [ 1744.636095] Key type ._llcrypt unregistered [ 1754.842179] Key type ._llcrypt registered [ 1754.843513] Key type .llcrypt registered [ 1756.192913] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:37:30 (1763347050) [ 1756.599150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.624363] alg: No test for adler32 (adler32-zlib) [ 1757.565867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.636987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.685944] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1758.343806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1758.423300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1764.339444] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:37:39 (1763347059) [ 1764.413064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.435591] LNet: 113559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.448860] LNetError: 113559:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.483801] LNet: Removed LNI 192.168.201.29@tcp [ 1764.513530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.563594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1764.589376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1769.913759] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:37:44 (1763347064) [ 1770.062506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.103226] LNet: 114147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.140624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.201211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1770.522178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1770.545866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1770.592392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1771.069093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1771.112433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1771.224636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.255155] LNet: 114181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.287137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.357698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1771.864698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1771.891542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1771.939561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1772.406037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1772.465516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1777.871464] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:37:52 (1763347072) [ 1778.065022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.114523] LNet: 114780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.163910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.207712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.234358] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1778.956862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1778.978186] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1779.655190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1779.707391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1779.922640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.946990] LNetError: 115061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.959091] LNet: Removed LNI 192.168.201.29@tcp [ 1779.977992] LNet: Removed LNI 192.168.201.29@tcp2 [ 1780.001670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.055719] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1780.060286] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1780.064681] LNet: 115067:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.462527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1780.661897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.731404] LNet: Removed LNI 192.168.201.29@tcp [ 1781.734155] LNetError: 115104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.748302] LNetError: 115104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1781.765419] LNet: Removed LNI 192.168.201.29@tcp2 [ 1781.797926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.861671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.898122] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1782.664626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1782.683536] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1783.451101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1783.485052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1783.583518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1783.786543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.817588] LNet: 115389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.822081] LNet: 115389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1783.828714] LNetError: 115389:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.833394] LNet: Removed LNI 192.168.201.29@tcp [ 1784.869996] LNet: Removed LNI 192.168.201.29@tcp2 [ 1784.886429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.937280] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1784.945162] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1784.949567] LNet: 115395:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1785.222284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1785.426421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.455557] LNet: Removed LNI 192.168.201.29@tcp [ 1786.472780] LNet: Removed LNI 192.168.201.29@tcp2 [ 1786.533563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.680109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.730413] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1787.691232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1787.717472] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1788.831071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1788.889425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1789.180455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.216137] LNetError: 115712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.227191] LNetError: 115712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1789.229907] LNet: Removed LNI 192.168.201.29@tcp [ 1789.252333] LNet: Removed LNI 192.168.201.29@tcp2 [ 1789.290427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.398495] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1789.411538] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1789.415703] LNet: 115718:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.337804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1790.651102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.712325] LNet: Removed LNI 192.168.201.29@tcp [ 1791.720712] LNet: Removed LNI 192.168.201.29@tcp2 [ 1791.799638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.919492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.972735] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1792.717581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1792.748472] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1793.687416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1793.734086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1793.782691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1794.010642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.025073] LNet: 116039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.037933] LNet: 116039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1795.107096] LNet: Removed LNI 192.168.201.29@tcp [ 1796.133718] LNet: Removed LNI 192.168.201.29@tcp2 [ 1796.201928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.330987] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1796.345553] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1796.361318] LNet: 116046:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1797.159259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1797.350008] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.392605] LNetError: 116083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.397691] LNetError: 116083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1797.410223] LNet: Removed LNI 192.168.201.29@tcp [ 1798.440094] LNet: Removed LNI 192.168.201.29@tcp2 [ 1798.457716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.585107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.635165] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1799.871310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1799.932914] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1800.949209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1800.984837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1801.034470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1801.488219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.527392] LNet: Removed LNI 192.168.201.29@tcp [ 1802.544668] LNet: Removed LNI 192.168.201.29@tcp2 [ 1802.583263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.662120] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1802.665592] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1802.671476] LNet: 116386:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.126487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1803.609778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.639151] LNet: Removed LNI 192.168.201.29@tcp [ 1804.655984] LNet: Removed LNI 192.168.201.29@tcp2 [ 1804.679566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.756476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.817715] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1805.584467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1805.610357] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1806.516549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1806.552895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1806.578949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1806.630064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1806.932031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.031173] LNet: Removed LNI 192.168.201.29@tcp [ 1809.069110] LNet: Removed LNI 192.168.201.29@tcp2 [ 1809.097141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.190850] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1809.208530] LNet: Added LNI 192.168.201.29@tcp2 [8/256/0/180] [ 1809.625524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1815.533151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1817.002919] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:38:31 (1763347111) [ 1817.220437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.242603] LNet: 117505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.245537] LNet: 117505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1817.250916] LNetError: 117505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.254538] LNetError: 117505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1817.260491] LNet: Removed LNI 192.168.201.29@tcp [ 1818.282703] LNet: Removed LNI 192.168.201.29@tcp2 [ 1818.313926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.642665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.668897] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1819.348854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1819.368795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1820.080673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.119507] LNet: Removed LNI 192.168.201.29@tcp [ 1820.129844] LNet: Removed LNI 10.1.2.3@tcp [ 1820.157489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.239327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1820.291980] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1820.299566] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1820.600488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1820.632802] LNet: Removed LNI 192.168.201.29@tcp [ 1820.643709] LNet: Removed LNI 10.1.2.3@tcp [ 1826.224036] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:38:41 (1763347121) [ 1826.408867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.498906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.595196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1826.673325] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1826.676622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1826.680234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1829.134755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.194133] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1829.967317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1829.990626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1831.749495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.29@tcp [ 1831.783927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1831.856493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.29@tcp -m PUT -e local_timeout -r 1 [ 1831.948085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1832.003771] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1832.011957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1832.244623] LNetError: 118448:0:(lib-move.c:4947:lnet_parse()) 192.168.201.29@tcp, src 192.168.201.29@tcp: Dropping PUT (error -108 looking up sender) [ 1834.272570] LNet: Removed LNI 192.168.201.29@tcp [ 1834.285073] LNet: Removed LNI 10.1.2.3@tcp [ 1834.669853] Key type .llcrypt unregistered [ 1834.673435] Key type ._llcrypt unregistered [ 1843.236800] Key type ._llcrypt registered [ 1843.239142] Key type .llcrypt registered [ 1844.481516] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:38:59 (1763347139) [ 1844.799884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.821440] alg: No test for adler32 (adler32-zlib) [ 1845.760326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.097278] LNet: 119754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.708227] Key type .llcrypt unregistered [ 1846.710255] Key type ._llcrypt unregistered [ 1851.200534] Key type ._llcrypt registered [ 1851.203033] Key type .llcrypt registered [ 1851.351492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1851.363823] alg: No test for adler32 (adler32-zlib) [ 1852.225586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1852.245785] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1858.257958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1858.413840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1858.987624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.129@tcp [ 1859.211470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1859.241357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1859.282688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1859.322364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1859.346748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1859.369575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1859.395734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.426314] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.129@tcp [ 1859.450863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.486482] LNet: 120199:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.536361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1859.579717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1859.611182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1859.638293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1859.664780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1859.727758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1860.349827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1860.534605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1860.563361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1860.589375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1860.613392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1860.643834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1860.664664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1860.701058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1860.721413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1860.740848] LNet: 120317:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1860.759092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1860.779578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1860.801888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1860.821351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1860.860974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1860.911935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1861.452574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.129@tcp [ 1861.631268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1861.659081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1861.682502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1861.728698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1861.756549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1861.796923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1861.817891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1861.839496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1861.848660] LNet: 120434:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1861.859745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1861.909246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1861.944737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1861.965825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1861.996905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1862.018596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1862.553572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1862.791467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1862.847208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1862.879553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1862.939659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1862.993962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1863.029541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1863.056774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1863.083851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1863.110269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1863.133893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1863.163537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1863.185111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1863.223104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1863.293960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1863.573784] LNet: 120596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.580640] LNetError: 120596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.615619] LNet: Removed LNI 192.168.201.29@tcp [ 1864.992839] Key type .llcrypt unregistered [ 1864.995184] Key type ._llcrypt unregistered [ 1879.918187] Key type ._llcrypt registered [ 1879.922422] Key type .llcrypt registered [ 1881.301126] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:39:35 (1763347175) [ 1881.784228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.808578] alg: No test for adler32 (adler32-zlib) [ 1882.805621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.155281] LNet: 121481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.795366] Key type .llcrypt unregistered [ 1883.797060] Key type ._llcrypt unregistered [ 1889.253033] Key type ._llcrypt registered [ 1889.265521] Key type .llcrypt registered [ 1889.468404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.482760] alg: No test for adler32 (adler32-zlib) [ 1890.434572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1890.466401] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1897.679611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.129@tcp [ 1905.550841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.579790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1905.610220] LNet: Added LNI 192.168.201.29@tcp1 [8/256/0/180] [ 1906.448931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1907.050097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.129@tcp [ 1907.250781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1907.288736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1907.334128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1907.390419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1907.443175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1907.496639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1907.530289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.566807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1907.589420] LNet: 122130:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1907.605823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1907.664381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1907.714920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1907.753830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1907.784153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1907.804612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1908.499066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1908.741458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1908.770426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1908.799551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1908.828237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1908.867962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1908.891777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1908.897395] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.129@tcp, 192.168.201.129@tcp1 [ 1908.933977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1908.981653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1909.010204] LNet: 122247:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1909.048727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1909.103156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1909.168544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1909.229825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1909.288740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1909.330709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1910.111571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.129@tcp [ 1910.365925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1910.389382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1910.418872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1910.448914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1910.488469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1910.523235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1910.572831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1910.608995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1910.629649] LNet: 122364:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1910.667904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1910.705551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1910.741691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1910.770929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1910.806907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1910.849544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1911.443722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1911.676517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1911.715267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1911.738600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1911.763758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1911.793978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1911.831462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1911.883572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1911.923323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1911.960461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1911.988379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1912.025531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1912.064044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1912.098509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1912.121731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1916.374066] LNet: 122561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.402114] LNetError: 122561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.411291] LNet: Removed LNI 192.168.201.29@tcp [ 1916.427429] LNet: Removed LNI 192.168.201.29@tcp1 [ 1916.826770] Key type .llcrypt unregistered [ 1916.829215] Key type ._llcrypt unregistered [ 1933.008137] Key type ._llcrypt registered [ 1933.010276] Key type .llcrypt registered [ 1934.621713] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1757 sec ========== 21:40:28 (1763347228) [ 1935.362914] Key type .llcrypt unregistered [ 1935.366923] Key type ._llcrypt unregistered [ 1947.964987] Key type ._llcrypt registered [ 1947.969726] Key type .llcrypt registered [ 1948.515835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1948.539888] alg: No test for adler32 (adler32-zlib) [ 1949.764573] Lustre: Lustre: Build Version: 2.16.61_45_g58940e0 [ 1950.094295] LNet: Added LNI 192.168.201.29@tcp [8/256/0/180] [ 1951.823207] Key type lgssc registered [ 1953.282673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.051303] Lustre: Mounted lustre-client [ 2010.474270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2022.098733] LustreError: 125785:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9bf608b5a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2022.120714] LustreError: 125785:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2022.152202] Lustre: Unmounted lustre-client [ 2052.418370] Key type lgssc unregistered [ 2052.717830] LNet: 126266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.736556] LNetError: 126266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.830278] LNet: Removed LNI 192.168.201.29@tcp [ 2053.447816] Key type .llcrypt unregistered [ 2053.450568] Key type ._llcrypt unregistered