[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 745923857 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003000] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.011000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011043] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012014] pid_max: default: 32768 minimum: 301 [ 0.013162] LSM: Security Framework initializing [ 0.014080] Yama: becoming mindful. [ 0.015059] SELinux: Initializing. [ 0.017048] *** VALIDATE selinux *** [ 0.033895] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.043292] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.044165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.045133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.047130] *** VALIDATE tmpfs *** [ 0.050075] *** VALIDATE proc *** [ 0.051288] *** VALIDATE cgroup *** [ 0.052018] *** VALIDATE cgroup2 *** [ 0.053354] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.054150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.055012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.056030] Spectre V2 : User space: Vulnerable [ 0.057012] Speculative Store Bypass: Vulnerable [ 0.061028] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.063304] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.065124] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.066027] ... version: 2 [ 0.067016] ... bit width: 48 [ 0.068020] ... generic registers: 4 [ 0.069021] ... value mask: 0000ffffffffffff [ 0.070024] ... max period: 00007fffffffffff [ 0.071017] ... fixed-purpose events: 3 [ 0.072017] ... event mask: 000000070000000f [ 0.073392] rcu: Hierarchical SRCU implementation. [ 0.075896] smp: Bringing up secondary CPUs ... [ 0.076965] x86: Booting SMP configuration: [ 0.077028] .... node #0, CPUs: #1 #2 #3 [ 0.098016] smp: Brought up 1 node, 4 CPUs [ 0.100017] smpboot: Max logical packages: 1 [ 0.101016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.162444] node 0 deferred pages initialised in 57ms [ 0.168123] devtmpfs: initialized [ 0.170105] x86/mm: Memory block size: 128MB [ 0.176023] gcov: version magic: 0x41383552 [ 0.180715] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.181084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183120] pinctrl core: initialized pinctrl subsystem [ 0.184854] [ 0.185018] ************************************************************* [ 0.186015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187020] ** ** [ 0.188022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189020] ** ** [ 0.190019] ** This means that this kernel is built to expose internal ** [ 0.191290] ** IOMMU data structures, which may compromise security on ** [ 0.192019] ** your system. ** [ 0.193022] ** ** [ 0.194085] ** If you see this message and you are not debugging the ** [ 0.195016] ** kernel, report this immediately to your vendor! ** [ 0.196021] ** ** [ 0.197016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198015] ************************************************************* [ 0.202033] NET: Registered protocol family 16 [ 0.204714] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207988] cpuidle: using governor menu [ 0.209129] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210853] PCI: Using configuration type 1 for base access [ 0.214311] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238298] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243074] cryptd: max_cpu_qlen set to 1000 [ 0.248019] ACPI: Added _OSI(Module Device) [ 0.250024] ACPI: Added _OSI(Processor Device) [ 0.253020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.255016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.262938] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.274407] ACPI: Interpreter enabled [ 0.275076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.276014] ACPI: Using IOAPIC for interrupt routing [ 0.277120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.278572] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.293720] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.294051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.295027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.296103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299601] acpiphp: Slot [2] registered [ 0.300200] acpiphp: Slot [5] registered [ 0.301171] acpiphp: Slot [6] registered [ 0.302879] acpiphp: Slot [3] registered [ 0.303196] acpiphp: Slot [4] registered [ 0.304163] acpiphp: Slot [7] registered [ 0.306163] acpiphp: Slot [8] registered [ 0.307170] acpiphp: Slot [9] registered [ 0.308125] acpiphp: Slot [10] registered [ 0.310138] acpiphp: Slot [11] registered [ 0.311126] acpiphp: Slot [12] registered [ 0.312593] acpiphp: Slot [13] registered [ 0.314052] acpiphp: Slot [14] registered [ 0.316080] acpiphp: Slot [15] registered [ 0.318111] acpiphp: Slot [16] registered [ 0.320086] acpiphp: Slot [17] registered [ 0.321144] acpiphp: Slot [18] registered [ 0.322168] acpiphp: Slot [19] registered [ 0.323154] acpiphp: Slot [20] registered [ 0.324194] acpiphp: Slot [21] registered [ 0.325511] acpiphp: Slot [22] registered [ 0.326133] acpiphp: Slot [23] registered [ 0.327500] acpiphp: Slot [24] registered [ 0.328141] acpiphp: Slot [25] registered [ 0.329150] acpiphp: Slot [26] registered [ 0.330140] acpiphp: Slot [27] registered [ 0.331128] acpiphp: Slot [28] registered [ 0.332124] acpiphp: Slot [29] registered [ 0.333149] acpiphp: Slot [30] registered [ 0.334127] acpiphp: Slot [31] registered [ 0.335210] PCI host bridge to bus 0000:00 [ 0.336800] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342381] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345034] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348298] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354855] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.357055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360244] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362637] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364176] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368683] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.376024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383178] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.402552] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404937] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.407028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.414022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.422287] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.423802] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.424726] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.425561] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.426361] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.444176] iommu: Default domain type: Passthrough [ 0.446000] SCSI subsystem initialized [ 0.449657] ACPI: bus type USB registered [ 0.456102] usbcore: registered new interface driver usbfs [ 0.462152] usbcore: registered new interface driver hub [ 0.465132] usbcore: registered new device driver usb [ 0.470243] pps_core: LinuxPPS API ver. 1 registered [ 0.474013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.479083] PTP clock support registered [ 0.483051] EDAC MC: Ver: 3.0.0 [ 0.488196] PCI: Using ACPI for IRQ routing [ 0.494797] NetLabel: Initializing [ 0.497016] NetLabel: domain hash size = 128 [ 0.502017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.507092] NetLabel: unlabeled traffic allowed by default [ 0.513044] vgaarb: loaded [ 0.515035] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.517014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.525000] clocksource: Switched to clocksource kvm-clock [ 0.711667] VFS: Disk quotas dquot_6.6.0 [ 0.713253] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.715753] *** VALIDATE ramfs *** [ 0.716838] *** VALIDATE hugetlbfs *** [ 0.720357] pnp: PnP ACPI init [ 0.722979] pnp: PnP ACPI: found 6 devices [ 0.740169] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.743418] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.745670] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.747738] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.750238] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.753136] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.756163] NET: Registered protocol family 2 [ 0.759242] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.764969] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.768578] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.774128] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.777738] TCP: Hash tables configured (established 65536 bind 65536) [ 0.781127] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.784629] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.787496] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.790726] NET: Registered protocol family 1 [ 0.796622] RPC: Registered named UNIX socket transport module. [ 0.800616] RPC: Registered udp transport module. [ 0.802593] RPC: Registered tcp transport module. [ 0.804720] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.807578] NET: Registered protocol family 44 [ 0.809563] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.814652] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.818581] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.822987] PCI: CLS 0 bytes, default 64 [ 0.824984] Unpacking initramfs... [ 4.081584] debug: unmapping init [mem 0xffff8c7c3cc64000-0xffff8c7c3ffcffff] [ 4.087191] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.090788] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.097482] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.014795] Initialise system trusted keyrings [ 7.019302] Key type blacklist registered [ 7.026810] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.038971] zbud: loaded [ 7.048350] *** VALIDATE nfs *** [ 7.052149] *** VALIDATE nfs4 *** [ 7.066800] pstore: using deflate compression [ 7.083551] Platform Keyring initialized [ 7.454189] NET: Registered protocol family 38 [ 7.456670] Key type asymmetric registered [ 7.461624] Asymmetric key parser 'x509' registered [ 7.472971] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.488618] io scheduler mq-deadline registered [ 7.497423] io scheduler kyber registered [ 7.503331] io scheduler bfq registered [ 7.513566] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.518534] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.528978] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.539861] ACPI: Power Button [PWRF] [ 7.570248] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.595778] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.654272] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.710017] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.759483] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.770404] Non-volatile memory driver v1.3 [ 7.772780] Linux agpgart interface v0.103 [ 7.866424] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 7.870843] vda: detected capacity change from 0 to 76787712 [ 7.930126] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.947707] vdb: detected capacity change from 0 to 1073741824 [ 7.978161] libphy: Fixed MDIO Bus: probed [ 8.026760] usbcore: registered new interface driver usbserial_generic [ 8.036543] usbserial: USB Serial support registered for generic [ 8.044494] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.056546] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.061442] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.069287] mousedev: PS/2 mouse device common for all mice [ 8.086895] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.088181] rtc_cmos 00:05: RTC can wake from S4 [ 8.133140] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.138040] rtc_cmos 00:05: registered as rtc0 [ 8.164662] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.169621] intel_pstate: CPU model not supported [ 8.176512] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.188543] hid: raw HID events driver (C) Jiri Kosina [ 8.198947] usbcore: registered new interface driver usbhid [ 8.208495] usbhid: USB HID core driver [ 8.215443] drop_monitor: Initializing network drop monitor service [ 8.229640] Initializing XFRM netlink socket [ 8.236230] NET: Registered protocol family 10 [ 8.246129] Segment Routing with IPv6 [ 8.248471] NET: Registered protocol family 17 [ 8.263466] mpls_gso: MPLS GSO support [ 8.276202] RAS: Correctable Errors collector initialized. [ 8.279226] AVX version of gcm_enc/dec engaged. [ 8.284076] AES CTR mode by8 optimization enabled [ 8.466835] sched_clock: Marking stable (8466810731, 0)->(10447556900, -1980746169) [ 8.482735] registered taskstats version 1 [ 8.487986] Loading compiled-in X.509 certificates [ 8.498924] zswap: loaded using pool lzo/zbud [ 8.599278] Key type big_key registered [ 8.623299] Key type encrypted registered [ 8.625958] ima: No TPM chip found, activating TPM-bypass! [ 8.630766] ima: Allocated hash algorithm: sha1 [ 8.633254] ima: No architecture policies found [ 8.639979] evm: Initialising EVM extended attributes: [ 8.646933] evm: security.selinux [ 8.653656] evm: security.ima [ 8.657344] evm: security.capability [ 8.658807] evm: HMAC attrs: 0x1 [ 8.661763] rtc_cmos 00:05: setting system clock to 2026-09-07 20:27:40 UTC (1788812860) [ 8.676450] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 8.684941] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 8.695456] Write protecting the kernel read-only data: 28672k [ 8.709787] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 8.721480] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 8.800381] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.814587] systemd[1]: Detected virtualization kvm. [ 8.820478] systemd[1]: Detected architecture x86-64. [ 8.824920] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.873227] systemd[1]: No hostname configured. [ 8.876544] systemd[1]: Set hostname to . [ 8.880989] random: systemd: uninitialized urandom read (16 bytes read) [ 8.887172] systemd[1]: Initializing machine ID from random generator. [ 9.060182] random: ln: uninitialized urandom read (6 bytes read) [ 9.430041] random: systemd: uninitialized urandom read (16 bytes read) [ 9.437676] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.468708] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 9.479580] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ 11.528575] hrtimer: interrupt took 3558914 ns [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.493498] device-mapper: uevent: version 1.0.3 [ 12.506214] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 15.344500] random: fast init done [ 15.454476] virtio_net virtio0 ens2: renamed from eth0 [ 15.745366] scsi host0: ata_piix [ 16.047088] scsi host1: ata_piix [ 16.053233] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 16.057932] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 21.242444] random: crng init done [ 21.261812] random: 7 urandom warning(s) missed due to ratelimiting [ 24.396225] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 27.613365] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 31.599897] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.667754] SELinux: Disabled at runtime. [ 32.768551] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.788256] systemd[1]: Detected virtualization kvm. [ 32.791283] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.843566] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.851350] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.872807] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.889518] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.894638] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.905478] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.952114] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 35.724396] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 37.438629] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 38.792538] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.944955] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.231354] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.402320] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 43.995481] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (9s / no limit)[ 44.666719] NFS: Registering the id_resolver key type [ 44.669405] Key type id_resolver registered [ 44.672405] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-client login: [ 94.502687] libcfs: loading out-of-tree module taints kernel. [ 94.755202] Key type ._llcrypt registered [ 94.757039] Key type .llcrypt registered [ 95.145488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.155975] alg: No test for adler32 (adler32-zlib) [ 96.405781] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 97.011115] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 98.719300] Key type lgssc registered [ 99.999461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.485686] Lustre: Mounted lustre-client - version 2.17.58_41_g50dbf09 [ 189.301644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 201.593399] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing check_logdir /tmp/testlogs/ [ 210.214642] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing yml_node [ 211.432586] Lustre: lustre-OST0000-osc-ffff8c7c8b310000: disconnect after 24s idle [ 217.885122] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 221.539083] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 224.229915] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 226.322629] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 16:31:16 EDT 2026 [ 254.632492] Lustre: Unmounted lustre-client [ 286.232527] Key type lgssc unregistered [ 286.538723] LNet: 8414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.549285] LNetError: 8414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.572850] LNet: Removed LNI 192.168.204.59@tcp [ 287.673173] Key type .llcrypt unregistered [ 287.675115] Key type ._llcrypt unregistered [ 318.756316] Key type ._llcrypt registered [ 318.765648] Key type .llcrypt registered [ 318.951392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 318.972677] alg: No test for adler32 (adler32-zlib) [ 319.948655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 320.048860] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 320.069254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 324.079832] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 324.350438] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.367723] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.384518] LNet: Removed LNI 192.168.204.59@tcp [ 325.147618] Key type .llcrypt unregistered [ 325.153963] Key type ._llcrypt unregistered [ 332.158654] Key type ._llcrypt registered [ 332.160630] Key type .llcrypt registered [ 333.562505] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:33:03 (1788813183) [ 334.010847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 334.042079] alg: No test for adler32 (adler32-zlib) [ 334.992617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.174960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.221844] LNet: 9702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.273323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.359900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 335.377286] LNet: 9709:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 342.765688] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:33:13 (1788813193) [ 342.916634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.934741] LNet: 10284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.962609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.080957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 350.240829] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:33:20 (1788813200) [ 350.472337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.516477] LNet: 10875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.539570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.668490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 357.487322] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:33:28 (1788813208) [ 357.686372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.723809] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.768200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.869862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 365.526745] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:33:35 (1788813215) [ 365.707858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.774704] LNet: 12040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.824827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.981856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 366.554500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 373.978050] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:33:44 (1788813224) [ 374.143862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.168073] LNet: 12635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.207925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.357539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 375.173184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 375.739743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 383.753438] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:33:53 (1788813233) [ 384.129719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.185465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.279986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 392.717460] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:34:02 (1788813242) [ 392.850226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.917086] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.926749] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.955750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.053679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.082829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.654102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 393.701511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.471813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.519599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.451079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.564747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.195898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 396.247744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 397.025491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 397.078484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.871441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 397.952748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.801965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 398.902327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.757931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 399.841477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.832237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 400.897421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.364353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 401.402885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 402.339693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 402.464382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 403.291558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 403.347628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.386413] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:34:22 (1788813262) [ 412.707449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.804737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.917618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.961299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 420.105148] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:34:30 (1788813270) [ 420.459215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.521597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.613664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.670733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 428.137330] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:34:38 (1788813278) [ 428.281443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.326566] LNet: 15667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.338308] LNet: 15667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 428.395884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.551930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 428.588935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 436.225375] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:34:46 (1788813286) [ 436.548058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.591822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.669776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 436.749314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.412258] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:34:54 (1788813294) [ 444.518919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.588307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.731658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 444.783051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 453.021607] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:35:02 (1788813302) [ 453.328922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.415892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.593624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.659657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 461.431588] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:35:11 (1788813311) [ 461.767210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.811272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.899783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.491864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.715251] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:35:20 (1788813320) [ 469.981804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.031098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.333181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.708745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.642567] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:35:29 (1788813329) [ 479.765811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.850829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.066726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.325314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.784957] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:35:39 (1788813339) [ 489.015481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.075713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.586597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.833383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.520650] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:35:47 (1788813347) [ 497.637338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.655505] LNet: 20501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.667724] LNet: 20501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 497.726861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.907252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.018762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.871940] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:35:55 (1788813355) [ 506.031410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.078874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.199970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.282894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.198858] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:36:04 (1788813364) [ 514.682139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.738611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.853080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 522.773571] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:36:13 (1788813373) [ 523.023246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.080962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.157460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 529.714453] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:36:20 (1788813380) [ 529.960901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.008974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.133976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 530.760634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 539.008368] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:36:28 (1788813388) [ 539.253313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.314454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.424671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 539.484813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 547.468191] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:36:37 (1788813397) [ 547.740359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.844175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.990683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 548.028391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 556.414114] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:36:46 (1788813406) [ 556.573893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.637436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.790058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 556.880725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 565.898979] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:36:55 (1788813415) [ 565.993155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.083688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.194305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 566.250101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 573.553240] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:37:03 (1788813423) [ 573.687880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.745779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.825272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 573.908942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 573.932881] LNetError: 25823:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 573.983461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 574.030683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 580.868716] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:37:11 (1788813431) [ 580.999543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.116767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.258342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qmTuvmfFQh [ 581.336953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qmTuvmfFQh [ 581.383934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qmTuvmfFQh [ 588.710893] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:37:19 (1788813439) [ 588.915446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.953574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.020232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 589.078493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 589.140112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 596.223799] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:37:26 (1788813446) [ 596.549264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.589647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.700543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.736497] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 598.261606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 599.683314] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 601.679276] Key type lgssc registered [ 602.893474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 606.516773] LNetError: 27585:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.59@tcp99 has no available nets [ 609.440254] Key type lgssc unregistered [ 609.840537] LNetError: 28689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.918684] LNet: Removed LNI 192.168.204.59@tcp [ 611.717081] Key type .llcrypt unregistered [ 611.722473] Key type ._llcrypt unregistered [ 619.909663] Key type ._llcrypt registered [ 619.915452] Key type .llcrypt registered [ 621.464655] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:37:51 (1788813471) [ 621.935636] Key type .llcrypt unregistered [ 621.940466] Key type ._llcrypt unregistered [ 629.082590] Key type ._llcrypt registered [ 629.085419] Key type .llcrypt registered [ 629.261312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 629.277902] alg: No test for adler32 (adler32-zlib) [ 630.216193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.282920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.304898] LNet: 29599:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.312954] LNet: 29599:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.319870] LNet: 29599:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.334708] LNet: Message forwarding will be enabled in 2s [ 633.439692] LNet: Message forwarding enabled [ 633.933059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 633.981662] LNet: Message forwarding will stop in 2s to 6s [ 637.550948] LNet: Message forwarding disabled [ 638.587779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 638.624125] LNet: 29697:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 638.631748] LNet: 29697:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 638.638570] LNet: 29697:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 638.659596] LNet: Message forwarding will be enabled in 2s [ 638.882574] LNet: 29710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.732665] Key type .llcrypt unregistered [ 639.738443] Key type ._llcrypt unregistered [ 655.017229] Key type ._llcrypt registered [ 655.024244] Key type .llcrypt registered [ 656.756390] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:38:26 (1788813506) [ 657.347325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.411158] alg: No test for adler32 (adler32-zlib) [ 658.409413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.496379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 667.908851] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:38:38 (1788813518) [ 668.170565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.218104] LNet: 31109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.274916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.366656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 668.404626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 668.472272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 668.555991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 668.653255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 668.705400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 668.762322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 668.893179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 669.001511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 669.104742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 669.145372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 669.172911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 669.211606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 669.252919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 669.305162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 669.365225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 669.416885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 669.462732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 669.505665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 669.527798] LNetError: 31155:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 677.649641] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:38:47 (1788813527) [ 678.182774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.252383] LNet: 31727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.298209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.406609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.760406] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:38:56 (1788813536) [ 687.676575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg459-server [ 689.687675] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:38:59 (1788813539) [ 689.951407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.970702] LNet: 32659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.988772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.032228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.051341] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 691.097804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[60-64]@tcp [ 691.159745] LNet: 32792:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 691.213714] LNetError: 32663:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.60@tcp is being used as a gateway but routing feature is not turned on [ 691.251267] LNetError: 32663:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.60@tcp has gone from up to down [ 694.241708] LNetError: 32793:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.63@tcp has gone from up to down [ 699.450737] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:39:09 (1788813549) [ 701.328367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg459-server [ 703.332786] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:39:13 (1788813553) [ 703.721768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.769286] LNet: 33728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.782737] LNetError: 33728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.802498] LNet: Removed LNI 192.168.204.59@tcp [ 703.843307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.905127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.935575] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 704.646380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[60-65/2]@tcp [ 704.678556] LNet: 33845:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 704.692087] LNet: 33845:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 704.711783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[60-65/2]@tcp [ 711.897266] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:39:22 (1788813562) [ 712.386158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.428869] LNet: 34437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.441897] LNetError: 34437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.481798] LNet: Removed LNI 192.168.204.59@tcp [ 712.516943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.709907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 712.820843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.886965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.983937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.036570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.088542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.179479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 713.267883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.313343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.447813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.529925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.559940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.619198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 713.718948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.764774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.854280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.983720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.016248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.054905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 714.132962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.171346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.229358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.298909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.331351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.379882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.456877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.488681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.537779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.616449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.655837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.718847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 714.799563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.849876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.927801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.475984] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:39:32 (1788813572) [ 724.574217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg459-server [ 726.297692] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:39:36 (1788813576) [ 728.649786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg459-server [ 730.654281] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:39:41 (1788813581) [ 731.006925] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.014529] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 731.714804] Key type .llcrypt unregistered [ 731.719991] Key type ._llcrypt unregistered [ 739.219106] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.642815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.659369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.666396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.468888] Key type ._llcrypt registered [ 742.470785] Key type .llcrypt registered [ 742.657307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.679733] alg: No test for adler32 (adler32-zlib) [ 743.556525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.620121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.642089] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 744.420922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 744.457851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.794233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 746.825956] LNet: Removed LNI 10.1.2.3@tcp [ 746.943459] LNet: 36557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.947544] LNetError: 36557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.996770] LNet: Removed LNI 192.168.204.59@tcp [ 747.608614] Key type .llcrypt unregistered [ 747.610432] Key type ._llcrypt unregistered [ 763.005345] Key type ._llcrypt registered [ 763.007354] Key type .llcrypt registered [ 764.234629] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:40:14 (1788813614) [ 765.028704] Key type .llcrypt unregistered [ 765.033638] Key type ._llcrypt unregistered [ 773.184909] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 773.512118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 773.517515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.521185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 776.262637] Key type ._llcrypt registered [ 776.264216] Key type .llcrypt registered [ 776.451377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 776.464324] alg: No test for adler32 (adler32-zlib) [ 777.419935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.478975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.505661] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 778.339201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.367964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.448586] LNet: 38042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.465845] LNetError: 38042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 781.481153] LNet: Removed LNI 10.1.2.3@tcp [ 781.495191] LNet: Removed LNI 192.168.204.59@tcp [ 782.231621] Key type .llcrypt unregistered [ 782.235641] Key type ._llcrypt unregistered [ 796.826724] Key type ._llcrypt registered [ 796.829768] Key type .llcrypt registered [ 798.378422] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:40:48 (1788813648) [ 799.428356] Key type .llcrypt unregistered [ 799.431899] Key type ._llcrypt unregistered [ 806.951506] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.189931] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.193131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.195752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.908245] Key type ._llcrypt registered [ 809.915144] Key type .llcrypt registered [ 810.141558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.155524] alg: No test for adler32 (adler32-zlib) [ 811.060829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.163931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 811.188106] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 813.299167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 813.313846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.479244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 815.507427] LNet: Removed LNI 10.1.2.3@tcp [ 815.523035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 815.557488] LNet: Removed LNI 10.1.2.31@tcp [ 815.578859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.610544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.411608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 816.469403] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 817.635429] LNet: 39958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.652291] LNetError: 39958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 817.677226] LNet: Removed LNI 10.1.2.31@tcp [ 817.708692] LNet: Removed LNI 10.1.2.3@tcp [ 818.407270] Key type .llcrypt unregistered [ 818.409925] Key type ._llcrypt unregistered [ 833.596724] Key type ._llcrypt registered [ 833.598310] Key type .llcrypt registered [ 835.168380] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:41:25 (1788813685) [ 835.793558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.819959] alg: No test for adler32 (adler32-zlib) [ 836.806708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.880567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.915213] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 837.895054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.940226] LNet: 40908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.960641] LNetError: 40908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.980151] LNet: Removed LNI 192.168.204.59@tcp [ 838.015655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.084957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.59@tcp [ 838.141617] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 838.775837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.59@tcp [ 838.805431] LNet: Removed LNI 192.168.204.59@tcp [ 838.860111] LNet: 41043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.384451] Key type .llcrypt unregistered [ 839.389298] Key type ._llcrypt unregistered [ 852.919321] Key type ._llcrypt registered [ 852.924957] Key type .llcrypt registered [ 854.489965] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:41:44 (1788813704) [ 856.868927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg459-server [ 858.519333] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:41:49 (1788813709) [ 859.503404] Key type .llcrypt unregistered [ 859.508912] Key type ._llcrypt unregistered [ 867.855991] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.158475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.165144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.174252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.910079] Key type ._llcrypt registered [ 870.912297] Key type .llcrypt registered [ 871.351482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.384918] alg: No test for adler32 (adler32-zlib) [ 872.476613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.558178] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 873.419189] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 875.747135] LNet: 42873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.758897] LNetError: 42873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.765133] LNet: Removed LNI 192.168.204.59@tcp1 [ 875.776968] LNet: Removed LNI 10.1.2.3@tcp2 [ 876.426756] Key type .llcrypt unregistered [ 876.429488] Key type ._llcrypt unregistered [ 891.255855] Key type ._llcrypt registered [ 891.259522] Key type .llcrypt registered [ 893.125036] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:42:22 (1788813742) [ 893.568620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.590266] alg: No test for adler32 (adler32-zlib) [ 894.592433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.709599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.742631] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 895.685895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.724410] LNet: 43842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.730215] LNetError: 43842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.749954] LNet: Removed LNI 192.168.204.59@tcp [ 895.771713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.848285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 895.898930] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 904.124689] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:42:34 (1788813754) [ 904.355570] LNet: 44432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.370530] LNetError: 44432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.395459] LNet: Removed LNI 192.168.204.59@tcp [ 905.019506] Key type .llcrypt unregistered [ 905.021489] Key type ._llcrypt unregistered [ 912.550269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 912.792509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.800367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.806251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.532147] Key type ._llcrypt registered [ 915.536313] Key type .llcrypt registered [ 915.791191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.811867] alg: No test for adler32 (adler32-zlib) [ 916.835080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.909592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.970875] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 918.162737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 918.198134] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.014358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.035244] LNet: 45169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.044086] LNetError: 45169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.051825] LNet: Removed LNI 10.1.2.3@tcp [ 921.063871] LNet: Removed LNI 192.168.204.59@tcp [ 921.089228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.167351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.200790] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 921.205202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.768893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.806193] LNet: 45196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.818797] LNetError: 45196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.832751] LNetError: 45196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 921.843280] LNet: Removed LNI 10.1.2.3@tcp [ 921.864187] LNet: Removed LNI 192.168.204.59@tcp [ 921.894207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.010626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 922.047738] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 922.059021] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.632557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 922.681311] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 924.364580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.405273] LNet: 45359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.411909] LNetError: 45359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.421596] LNetError: 45359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.445705] LNet: Removed LNI 10.1.2.3@tcp [ 924.447325] LNet: Removed LNI 192.168.204.59@tcp [ 924.474243] LNet: Removed LNI 192.168.204.59@tcp2 [ 924.518190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.599311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 924.630696] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 924.641579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.647756] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 933.547621] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:43:04 (1788813784) [ 933.801047] LNet: 45956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.813274] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.824528] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 933.829730] LNet: Removed LNI 10.1.2.3@tcp [ 933.832147] LNet: Removed LNI 192.168.204.59@tcp [ 933.845414] LNet: Removed LNI 192.168.204.59@tcp2 [ 934.647469] Key type .llcrypt unregistered [ 934.655468] Key type ._llcrypt unregistered [ 941.994359] Key type ._llcrypt registered [ 941.996796] Key type .llcrypt registered [ 942.209304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.229413] alg: No test for adler32 (adler32-zlib) [ 943.173697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.247688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.286042] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 944.046782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.079539] LNet: 46347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.090630] LNetError: 46347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.105011] LNet: Removed LNI 192.168.204.59@tcp [ 944.128918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.205794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.241774] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 944.548639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.571859] LNet: Removed LNI 192.168.204.59@tcp [ 944.589936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.644616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.662375] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 945.197554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.243489] LNet: 46408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.248583] LNet: 46408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 945.259454] LNetError: 46408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.265509] LNetError: 46408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 945.281816] LNet: Removed LNI 192.168.204.59@tcp [ 945.306502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.382838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.415613] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 945.884179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.102977] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.110914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.118852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.496983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.543236] LNet: 46484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.555527] LNetError: 46484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.589940] LNet: Removed LNI 192.168.204.59@tcp [ 948.649156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.738275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 948.781172] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 955.920217] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:43:26 (1788813806) [ 956.240498] LNet: 47081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.256225] LNetError: 47081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.276500] LNet: Removed LNI 192.168.204.59@tcp [ 956.873940] Key type .llcrypt unregistered [ 956.875748] Key type ._llcrypt unregistered [ 964.130429] Key type ._llcrypt registered [ 964.133219] Key type .llcrypt registered [ 964.470866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.495689] alg: No test for adler32 (adler32-zlib) [ 965.535030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.602839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.632993] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 966.714858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.748299] LNet: 47477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.762561] LNetError: 47477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.785265] LNet: Removed LNI 192.168.204.59@tcp [ 966.816931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.870992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 966.891399] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 974.442758] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:43:45 (1788813825) [ 974.760763] LNet: 48069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.773441] LNetError: 48069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.792887] LNet: Removed LNI 192.168.204.59@tcp [ 975.541499] Key type .llcrypt unregistered [ 975.545630] Key type ._llcrypt unregistered [ 982.151671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.413079] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.418623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.427866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 985.053296] Key type ._llcrypt registered [ 985.059272] Key type .llcrypt registered [ 985.254879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.271676] alg: No test for adler32 (adler32-zlib) [ 986.285802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.393057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.457081] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 987.404717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.431555] LNet: 48516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.442238] LNetError: 48516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.459918] LNet: Removed LNI 192.168.204.59@tcp [ 987.490275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.574581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 987.606324] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 994.997780] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:44:05 (1788813845) [ 995.286751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.320914] LNet: 49115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.325329] LNetError: 49115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.365531] LNet: Removed LNI 192.168.204.59@tcp [ 995.408344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.534674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1002.457156] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:44:12 (1788813852) [ 1002.888399] LNet: 49697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.543758] Key type .llcrypt unregistered [ 1003.556698] Key type ._llcrypt unregistered [ 1010.753864] Key type ._llcrypt registered [ 1010.756558] Key type .llcrypt registered [ 1011.030977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.070518] alg: No test for adler32 (adler32-zlib) [ 1012.037960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.146088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1012.209837] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1013.159721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.181628] LNet: 50091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.187837] LNetError: 50091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.206459] LNet: Removed LNI 192.168.204.59@tcp [ 1013.231463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.337705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1013.365580] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1021.401435] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:44:31 (1788813871) [ 1021.492507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1021.560749] LNet: 50677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.576129] LNetError: 50677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.616554] LNet: Removed LNI 192.168.204.59@tcp [ 1021.647787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.764436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1028.276593] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:44:38 (1788813878) [ 1029.990983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1031.777334] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:44:42 (1788813882) [ 1033.269630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1034.997631] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:44:45 (1788813885) [ 1036.622450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1038.516117] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:44:48 (1788813888) [ 1040.215679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1042.419881] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:44:52 (1788813892) [ 1044.107327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1046.218596] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:44:56 (1788813896) [ 1046.372837] LNet: 52702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.364728] Key type .llcrypt unregistered [ 1047.366508] Key type ._llcrypt unregistered [ 1056.243749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.559707] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1056.574031] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1056.596117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.399847] Key type ._llcrypt registered [ 1059.401822] Key type .llcrypt registered [ 1059.638286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.657943] alg: No test for adler32 (adler32-zlib) [ 1060.637559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.707855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.734470] LNet: Added LNI 192.168.204.59@tcp [8/512/0/180] [ 1060.739251] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1060.744528] LNet: 53026:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.867925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.883426] LNet: 53045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.889602] LNetError: 53045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.894679] LNet: Removed LNI 10.1.2.3@tcp [ 1060.897602] LNetError: 53045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.910651] LNet: Removed LNI 192.168.204.59@tcp [ 1060.947694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.015644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.064883] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1061.129476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.150563] LNet: 53069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.156721] LNetError: 53069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.176117] LNet: Removed LNI 192.168.204.59@tcp [ 1061.217768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.268390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.306440] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1061.312491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.320490] LNet: 53077:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1061.423651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.469924] LNet: 53095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.475453] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.484283] LNet: Removed LNI 10.1.2.3@tcp [ 1061.486769] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.512456] LNet: Removed LNI 192.168.204.59@tcp [ 1061.551708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.638489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1061.669740] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1061.680803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.684353] LNet: 53103:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1061.920498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.940486] LNet: 53129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.946092] LNetError: 53129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.954412] LNet: Removed LNI 10.1.2.3@tcp [ 1061.956701] LNetError: 53129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.969696] LNet: Removed LNI 192.168.204.59@tcp [ 1062.011525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.119805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1062.181454] LNet: Added LNI 192.168.204.59@tcp [8/256/0/182] [ 1062.201084] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1062.354387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.375570] LNet: 53157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.381392] LNetError: 53157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.386499] LNet: Removed LNI 10.1.2.3@tcp [ 1062.388536] LNetError: 53157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.398778] LNet: Removed LNI 192.168.204.59@tcp [ 1062.436485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.593779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1062.629443] LNet: Added LNI 192.168.204.59@tcp [8/256/0/182] [ 1062.640443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1062.642574] LNet: 53165:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1062.763152] LNet: Removed LNI 10.1.2.3@tcp [ 1062.772991] LNet: Removed LNI 192.168.204.59@tcp [ 1063.503774] Key type .llcrypt unregistered [ 1063.506048] Key type ._llcrypt unregistered [ 1075.708611] Key type ._llcrypt registered [ 1075.711638] Key type .llcrypt registered [ 1077.447703] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:45:27 (1788813927) [ 1079.154210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1081.217633] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:45:31 (1788813931) [ 1081.747653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.764653] alg: No test for adler32 (adler32-zlib) [ 1082.842114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.952376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1082.974465] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1083.980814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.59 [ 1083.996707] LNet: 54426:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1084.004559] LNet: Removed LNI 192.168.204.59@tcp [ 1090.688102] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:45:41 (1788813941) [ 1090.789523] LNet: 54988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.797722] LNetError: 54988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.818229] LNet: Removed LNI 192.168.204.59@tcp [ 1091.486404] Key type .llcrypt unregistered [ 1091.488645] Key type ._llcrypt unregistered [ 1098.698603] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.037763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.770375] Key type ._llcrypt registered [ 1101.773304] Key type .llcrypt registered [ 1101.965191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.976625] alg: No test for adler32 (adler32-zlib) [ 1103.517649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1110.217614] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:46:00 (1788813960) [ 1110.409349] LNet: 55920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.417049] LNetError: 55920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1110.450463] LNet: Removed LNI 10.1.2.3@tcp [ 1111.051108] Key type .llcrypt unregistered [ 1111.052814] Key type ._llcrypt unregistered [ 1118.105904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1118.454651] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1118.480474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1118.492932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1121.325322] Key type ._llcrypt registered [ 1121.328287] Key type .llcrypt registered [ 1121.679789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.719840] alg: No test for adler32 (adler32-zlib) [ 1122.750687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1129.982835] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:46:20 (1788813980) [ 1131.848143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1133.617528] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:46:23 (1788813983) [ 1133.926590] LNet: 57105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.931611] LNetError: 57105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1133.949709] LNet: Removed LNI 10.1.2.3@tcp [ 1134.848497] Key type .llcrypt unregistered [ 1134.857280] Key type ._llcrypt unregistered [ 1142.014453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1142.332726] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1142.337126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1142.340368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1145.090967] Key type ._llcrypt registered [ 1145.092752] Key type .llcrypt registered [ 1145.270599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.282604] alg: No test for adler32 (adler32-zlib) [ 1146.237758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.335758] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1156.739962] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:46:46 (1788814006) [ 1157.261576] LNet: 58332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.266978] LNetError: 58332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1157.283217] LNet: Removed LNI 10.1.2.3@tcp [ 1157.946563] Key type .llcrypt unregistered [ 1157.950398] Key type ._llcrypt unregistered [ 1165.264698] Key type ._llcrypt registered [ 1165.269209] Key type .llcrypt registered [ 1165.531524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1165.551473] alg: No test for adler32 (adler32-zlib) [ 1166.491669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1166.524636] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1175.396622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1176.270353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 1176.305692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 1176.348960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1176.369777] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1176.640216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.699895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.763592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.799145] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1176.826993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.885238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1176.916707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.949992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.023509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.072419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.095909] LNet: 58769:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.131698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.163522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.213961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.285989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.324377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.372371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.435906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.088139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 1178.123902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 1178.163698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1178.185418] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1178.423741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.454888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.483711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.509863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.533728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1178.579764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.620972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.669742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.718239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.741460] LNet: 58880:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.768475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.843665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.919488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.972890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.019100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.094630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.124440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1179.769779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 1179.818360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 1179.852341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1179.869699] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1180.120753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.146619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.169828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.198234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.223461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1180.255616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.280411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.304420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.329862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.343482] LNet: 58991:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.360958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.386611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.413101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.436289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.466389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.490286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.515416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.944586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 1180.974490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 1181.002296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1181.244793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.268804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.298765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.323790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.350607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.381686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.426310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.459880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.487880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.520868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.565750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.601507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.630901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.666437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.727270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.808501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.390990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 1182.436853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 1182.468266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1182.483832] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1182.488236] LNet: Skipped 1 previous similar message [ 1182.700678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.763236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.810056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.861319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.899969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1182.929676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.972843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.011947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.040925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.054214] LNet: 59213:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.058591] LNet: 59213:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1183.072898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.098783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.127035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.152892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.179882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.213272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.252312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.807473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 1183.848643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 1183.889369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1184.174189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.210902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.253655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.289421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.318707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1184.340352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.384947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.415814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.441617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.472636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.515578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.546749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.589290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.653403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.702742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.741334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.037927] LNet: 59366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.042990] LNetError: 59366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.065565] LNet: Removed LNI 192.168.204.59@tcp [ 1185.476581] Key type .llcrypt unregistered [ 1185.478419] Key type ._llcrypt unregistered [ 1203.826499] Key type ._llcrypt registered [ 1203.829483] Key type .llcrypt registered [ 1205.491503] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:47:35 (1788814055) [ 1206.308643] Key type .llcrypt unregistered [ 1206.312843] Key type ._llcrypt unregistered [ 1212.735842] Key type ._llcrypt registered [ 1212.738312] Key type .llcrypt registered [ 1212.879642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1212.890797] alg: No test for adler32 (adler32-zlib) [ 1213.806564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1213.840499] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1221.245776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1232.667253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.734421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1232.756687] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1234.118196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 1234.157784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 1 [ 1234.225819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 1234.275904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 1 [ 1234.332837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_interrupt -r 1 [ 1234.361992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 1 [ 1234.386899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_interrupt -r 1 [ 1234.435597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 1 [ 1234.492778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1234.515139] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1234.526654] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1234.771816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.810280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.850605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.886544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.929884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1234.972316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.006529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.040586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.081978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.099438] LNet: 60830:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.117425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.151985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.185990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.241581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.287750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.332743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.368479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.941821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 1235.985875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 1 [ 1236.023132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 1236.062386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 1 [ 1236.098406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_dropped -r 1 [ 1236.132061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 1 [ 1236.171729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_dropped -r 1 [ 1236.210722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 1 [ 1236.260938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1236.279236] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1236.284224] LNet: Skipped 2 previous similar messages [ 1236.503387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.531616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.556651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.586837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.613234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1236.637868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.665545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.717985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.753851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.784793] LNet: 60953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.821637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.849701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.879992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.914525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.947903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.992712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.031161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.621921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 1237.657872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 1 [ 1237.687866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 1237.728777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 1 [ 1237.762347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_aborted -r 1 [ 1237.792613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 1 [ 1237.846326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_aborted -r 1 [ 1237.875071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 1 [ 1237.913284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1237.932709] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1237.937554] LNet: Skipped 2 previous similar messages [ 1238.164096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.199511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.225679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.257590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.286886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1238.313896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.345823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.371957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.401706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.421560] LNet: 61076:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.448627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.482213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.522699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.558585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.586033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.617272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.645789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.244782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 1239.280981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 1 [ 1239.315349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 1239.371983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 1 [ 1239.450645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_no_route -r 1 [ 1239.522590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 1 [ 1239.593732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_no_route -r 1 [ 1239.648171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 1 [ 1239.720680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1239.964740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.004833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.048664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.077374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.130903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1240.163557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.191579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.224618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.254286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.281769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.325258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.360901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.394285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.431190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.490613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.522148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1241.009408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 1241.045692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 1 [ 1241.088598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 1241.146813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 1 [ 1241.218197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_timeout -r 1 [ 1241.271133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 1 [ 1241.313852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_timeout -r 1 [ 1241.349761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 1 [ 1241.390074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1241.408854] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1241.415573] LNet: Skipped 5 previous similar messages [ 1241.638202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.671748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.710734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.746845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.786974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.843173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.897717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.943369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.994689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.021379] LNet: 61323:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.030670] LNet: 61323:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1242.051077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.088129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.119503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.156985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.197414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.232444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.268143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.851158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 1242.877842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_error -r 1 [ 1242.911763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 1242.955693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_error -r 1 [ 1243.007670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_error -r 1 [ 1243.038763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_error -r 1 [ 1243.076908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_error -r 1 [ 1243.141413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_error -r 1 [ 1243.173185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1243.420435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.486388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.531501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.607553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.655741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.697530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.738548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.773672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.797995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.822450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.847964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.877486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.916268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.969903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.011604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.059577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1249.890422] LNet: 61522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.895754] LNetError: 61522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.905780] LNet: Removed LNI 192.168.204.59@tcp [ 1249.920623] LNet: Removed LNI 192.168.204.59@tcp1 [ 1250.425654] Key type .llcrypt unregistered [ 1250.431342] Key type ._llcrypt unregistered [ 1266.771522] Key type ._llcrypt registered [ 1266.776173] Key type .llcrypt registered [ 1268.131511] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:48:38 (1788814118) [ 1269.068654] Key type .llcrypt unregistered [ 1269.073123] Key type ._llcrypt unregistered [ 1276.164601] Key type ._llcrypt registered [ 1276.171816] Key type .llcrypt registered [ 1276.442624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.472323] alg: No test for adler32 (adler32-zlib) [ 1277.448659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.475549] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1285.645430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1286.446192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 1286.483380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 1286.544354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1286.577077] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1286.855619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.892979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.935603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.994523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.048126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1287.088936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.114168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.152365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.178721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.195781] LNet: 62769:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.217816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.249100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.276760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.303455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.337283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.368590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.401403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.996590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 1288.036577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 1288.097934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1288.129952] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1288.399657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.429619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.480123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.541085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.570954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.607863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.635633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.663594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.729141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.761388] LNet: 62880:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.796728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.866302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.897806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.931779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.958695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.990300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.021135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1289.493162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 1289.531783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 1289.586493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1289.624954] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1289.915785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.946654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.978983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.011934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.057061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1290.083124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.119339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.152805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.194149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.214946] LNet: 62991:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.233129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.281880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.321538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.362345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.412054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.458689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.498191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1290.759984] LNet: 63032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.770800] LNetError: 63032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.814366] LNet: Removed LNI 192.168.204.59@tcp [ 1292.309231] Key type .llcrypt unregistered [ 1292.312334] Key type ._llcrypt unregistered [ 1310.438203] Key type ._llcrypt registered [ 1310.441318] Key type .llcrypt registered [ 1311.931247] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:49:22 (1788814162) [ 1312.665691] Key type .llcrypt unregistered [ 1312.668337] Key type ._llcrypt unregistered [ 1318.960958] Key type ._llcrypt registered [ 1318.964647] Key type .llcrypt registered [ 1319.193544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.225602] alg: No test for adler32 (adler32-zlib) [ 1320.156433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.201373] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1327.705120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1336.986406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.042424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.066866] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1338.205955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 1338.236876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 1 [ 1338.270554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 1338.309705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 1 [ 1338.337256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_dropped -r 1 [ 1338.367989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 1 [ 1338.398356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_dropped -r 1 [ 1338.436371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 1 [ 1338.473565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1338.508499] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1338.517917] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 1338.770662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.810308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.874466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.940382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.990200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1339.027175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.085611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.117196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.154628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.182665] LNet: 64499:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.203832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.246762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.282730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.319481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.357948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.386145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.440954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1339.478134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1340.047977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 1340.102921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_error -r 1 [ 1340.144230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 1340.209952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_error -r 1 [ 1340.270257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_error -r 1 [ 1340.318332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_error -r 1 [ 1340.357668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_error -r 1 [ 1340.404505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_error -r 1 [ 1340.437130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1340.462942] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1340.469136] LNet: Skipped 2 previous similar messages [ 1340.742789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.785935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.824161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.871696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.900226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1340.949733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.011947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.061919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.102290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.146182] LNet: 64624:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.177584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.223507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.286154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.345225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.390121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.438084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.487091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.528812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.082539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 1342.113633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_timeout -r 1 [ 1342.158152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 1342.232686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_timeout -r 1 [ 1342.283140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_timeout -r 1 [ 1342.318799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_timeout -r 1 [ 1342.368974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_timeout -r 1 [ 1342.433321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_timeout -r 1 [ 1342.510067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.159@tcp [ 1342.571011] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1342.910558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.957929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1343.013141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1343.051639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.113929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1343.177948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.233111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.302544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.387930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.422660] LNet: 64750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.465837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.552657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.596065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.642030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.701545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.734535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.772294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.803590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.608897] LNet: 64828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.614986] LNetError: 64828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.665234] LNet: Removed LNI 192.168.204.59@tcp [ 1349.668064] LNetError: 64828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.687401] LNet: Removed LNI 192.168.204.59@tcp1 [ 1350.118531] Key type .llcrypt unregistered [ 1350.125741] Key type ._llcrypt unregistered [ 1365.277670] Key type ._llcrypt registered [ 1365.282208] Key type .llcrypt registered [ 1366.574706] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:50:16 (1788814216) [ 1367.936560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1369.638866] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:50:20 (1788814220) [ 1370.388500] Key type .llcrypt unregistered [ 1370.392995] Key type ._llcrypt unregistered [ 1376.798882] Key type ._llcrypt registered [ 1376.801333] Key type .llcrypt registered [ 1377.081600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1377.123569] alg: No test for adler32 (adler32-zlib) [ 1378.000294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1378.027211] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1385.471108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1386.310924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 1386.385900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 1386.437770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1386.469524] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1386.684630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1386.724211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1386.751468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1386.820465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1386.857991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1386.889656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1386.928159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1386.970396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1387.017855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1387.051386] LNet: 66369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1387.073309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1387.133199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1387.172750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1387.211991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.231430] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1387.235818] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1387.257719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1387.294887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1387.555749] LNet: 66408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.563152] LNetError: 66408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.588192] LNet: Removed LNI 192.168.204.59@tcp [ 1387.942741] Key type .llcrypt unregistered [ 1387.945761] Key type ._llcrypt unregistered [ 1406.614615] Key type ._llcrypt registered [ 1406.616372] Key type .llcrypt registered [ 1406.841001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.856486] alg: No test for adler32 (adler32-zlib) [ 1407.817755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1407.855913] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1415.120708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1424.014394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.108983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1424.158091] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1425.406275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 1425.438498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e network_timeout -r 1 [ 1425.472918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 1425.513358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e network_timeout -r 1 [ 1425.563092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e network_timeout -r 1 [ 1425.629781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e network_timeout -r 1 [ 1425.666572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e network_timeout -r 1 [ 1425.697831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e network_timeout -r 1 [ 1425.742524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1425.765661] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1426.011876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1426.040820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1426.065699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1426.091244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1426.116660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1426.144042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1426.172865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.199628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.223235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.241603] LNet: 67286:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.262449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.285458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.308549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.335336] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1426.337813] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 1426.339570] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1426.353643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.357161] LNet: Skipped 1 previous similar message [ 1426.394072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1426.417802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1430.607803] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1431.003597] LNet: 67360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.008237] LNetError: 67360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.018991] LNet: Removed LNI 192.168.204.59@tcp [ 1431.031554] LNet: Removed LNI 192.168.204.59@tcp1 [ 1431.503480] Key type .llcrypt unregistered [ 1431.508325] Key type ._llcrypt unregistered [ 1447.321653] Key type ._llcrypt registered [ 1447.323610] Key type .llcrypt registered [ 1449.070884] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:51:39 (1788814299) [ 1449.543107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.567393] alg: No test for adler32 (adler32-zlib) [ 1450.644396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.741774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.795259] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1451.782870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.825934] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1452.686714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1452.762853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1452.821376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1452.870469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1452.923770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1452.958774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.004852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 1453.032585] LNetError: 68459:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 1453.792450] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1453.794508] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1455.839425] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1456.576995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.938533] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1460.156820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1460.188804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1460.232522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.273103] LNet: 68567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.292335] LNetError: 68567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.306485] LNet: Removed LNI 192.168.204.59@tcp [ 1460.328755] LNet: Removed LNI 192.168.204.59@tcp1 [ 1460.377908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.468952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.496848] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1461.116614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1461.134561] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1461.874206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1461.909400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.981567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1462.016314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1462.049863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1462.085817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1462.116326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1462.156797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 1462.174561] LNetError: 68839:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 1462.495990] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1465.584693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1468.639589] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1468.646678] LNet: Skipped 1 previous similar message [ 1473.473575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1473.525531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1473.576956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.646853] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:52:10 (1788814330) [ 1480.782804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.838922] LNet: 69551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.851932] LNetError: 69551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.859629] LNetError: 69551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1480.867761] LNet: Removed LNI 192.168.204.59@tcp [ 1480.886531] LNet: Removed LNI 192.168.204.59@tcp1 [ 1480.927922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.013324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.043724] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1481.721296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.754506] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1482.714452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1482.803970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1482.849536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1482.896081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1482.949413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1483.005598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.062334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 0 [ 1483.176485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1484.064517] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.59@tcp [ 1484.064723] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1484.090178] LNet: Skipped 1 previous similar message [ 1498.735681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1498.882150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1498.919643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1498.959564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 500 [ 1499.077987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1505.636672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1505.684664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.703089] LNet: 70049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.708035] LNetError: 70049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.713498] LNetError: 70049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1505.718262] LNet: Removed LNI 192.168.204.59@tcp [ 1505.735576] LNet: Removed LNI 192.168.204.59@tcp1 [ 1505.758968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.801934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.819400] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1506.352607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.366762] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1507.034989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1507.069880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1507.152352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1507.179755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1507.223897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1507.265365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1507.314899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1507.346387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.59@tcp --health 0 [ 1508.833226] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1508.840609] LNet: Skipped 3 previous similar messages [ 1509.753675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1518.796648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1518.843483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1518.872195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.118081] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:52:55 (1788814375) [ 1525.550250] LNet: 71095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.564599] LNetError: 71095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.570128] LNetError: 71095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1525.578617] LNet: Removed LNI 192.168.204.59@tcp [ 1525.591087] LNet: Removed LNI 192.168.204.59@tcp1 [ 1526.179396] Key type .llcrypt unregistered [ 1526.183433] Key type ._llcrypt unregistered [ 1531.889458] Key type ._llcrypt registered [ 1531.895956] Key type .llcrypt registered [ 1532.051022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1532.067305] alg: No test for adler32 (adler32-zlib) [ 1533.107358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1533.155399] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1541.885434] Lustre: DEBUG MARKER: Initial discovery [ 1541.919671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 1544.051993] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1544.111758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1544.162827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 1544.189746] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1544.210925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1544.265762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.159@tcp -r 1 -m GET -l 3 [ 1544.306535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.159@tcp -r 1 -m GET -e local_error [ 1544.351844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.159@tcp -r 1 -m PUT -l 6 [ 1544.479711] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1545.534747] Lustre: DEBUG MARKER: Force oleg459-client.virtnet to discover 192.168.204.159@tcp (in background) [ 1545.617317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.159@tcp [ 1546.898630] Lustre: DEBUG MARKER: Force oleg459-server to discover 192.168.204.59@tcp [ 1548.639542] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1548.777281] Lustre: DEBUG MARKER: Wait for 71849 [ 1550.136683] Lustre: DEBUG MARKER: Finished wait on 71849 [ 1550.333150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1550.375157] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1550.402148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1550.673799] LNet: 72306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.680829] LNetError: 72306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.706859] LNet: Removed LNI 192.168.204.59@tcp [ 1551.070255] Key type .llcrypt unregistered [ 1551.073840] Key type ._llcrypt unregistered [ 1565.121438] Key type ._llcrypt registered [ 1565.125202] Key type .llcrypt registered [ 1566.447802] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:53:36 (1788814416) [ 1566.931051] Key type .llcrypt unregistered [ 1566.937815] Key type ._llcrypt unregistered [ 1572.592992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1572.765146] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1572.770314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1572.777217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1575.246962] Key type ._llcrypt registered [ 1575.249607] Key type .llcrypt registered [ 1575.451600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.464168] alg: No test for adler32 (adler32-zlib) [ 1576.381304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.470068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.506647] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1577.127537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1577.162035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1584.889165] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:53:55 (1788814435) [ 1585.041702] LNet: 74369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.061149] LNetError: 74369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1585.067083] LNet: Removed LNI 10.1.2.3@tcp [ 1585.090476] LNet: Removed LNI 192.168.204.59@tcp [ 1585.954859] Key type .llcrypt unregistered [ 1585.959259] Key type ._llcrypt unregistered [ 1592.935663] Key type ._llcrypt registered [ 1592.938670] Key type .llcrypt registered [ 1593.104951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.118592] alg: No test for adler32 (adler32-zlib) [ 1594.070612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.155335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.176572] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1594.850911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1594.892307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1595.853692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.59@tcp [ 1596.895182] LNetError: Unexpected error -22 connecting to 192.168.204.59@tcp at host 192.168.204.59:988 [ 1601.629646] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:54:12 (1788814452) [ 1601.738059] LNet: 75497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.752076] LNetError: 75497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1601.757945] LNet: Removed LNI 10.1.2.3@tcp [ 1601.780580] LNet: Removed LNI 192.168.204.59@tcp [ 1602.433577] Key type .llcrypt unregistered [ 1602.438063] Key type ._llcrypt unregistered [ 1607.781341] Key type ._llcrypt registered [ 1607.783497] Key type .llcrypt registered [ 1607.964399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1607.987656] alg: No test for adler32 (adler32-zlib) [ 1608.906862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.008571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1609.060360] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1609.577131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1609.591590] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 1610.449711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.59@tcp1 --nid 192.168.204.59@tcp2 [ 1614.291900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp1 192.168.204.59@tcp2 [ 1614.313736] LNetError: 76339:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.59@tcp1 to 192.168.204.59@tcp2 - Route table may be misconfigured [ 1614.335268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp2 192.168.204.59@tcp1 [ 1619.963401] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:54:30 (1788814470) [ 1620.108512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.141752] LNet: 76911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.157783] LNetError: 76911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.177974] LNet: Removed LNI 192.168.204.59@tcp1 [ 1620.214368] LNet: Removed LNI 192.168.204.59@tcp2 [ 1620.250538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.297953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.328604] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1620.891420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1620.910117] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1621.545930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 1621.581344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.59@tcp [ 1621.615456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp1 192.168.204.59@tcp1 [ 1621.655505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.59@tcp -d 192.168.204.59@tcp -e network_timeout [ 1621.703801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -e network_timeout [ 1621.736576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -e network_timeout [ 1621.782703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -e network_timeout [ 1621.810641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1621.825225] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1621.843509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1621.921354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1622.012535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1622.306506] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1627.779583] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:54:38 (1788814478) [ 1627.894271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.934408] LNet: 77770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.949709] LNetError: 77770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.954340] LNetError: 77770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1628.959618] LNet: Removed LNI 192.168.204.59@tcp [ 1628.962573] LNetError: 77770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.980706] LNet: Removed LNI 192.168.204.59@tcp1 [ 1629.017677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.099595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1629.315341] LNet: 77810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.678765] Key type .llcrypt unregistered [ 1629.681181] Key type ._llcrypt unregistered [ 1639.848603] Key type ._llcrypt registered [ 1639.852137] Key type .llcrypt registered [ 1641.192049] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:54:51 (1788814491) [ 1641.675595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.693902] alg: No test for adler32 (adler32-zlib) [ 1642.603809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.928231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1645.567708] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:54:56 (1788814496) [ 1645.846843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.875166] LNet: 78905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.898345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.967908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.002290] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1646.522208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1646.544442] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1647.512958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1647.552954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp1 [ 1647.599581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp1 [ 1653.070399] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:55:03 (1788814503) [ 1654.963490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg459-server [ 1656.829213] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:55:07 (1788814507) [ 1659.275875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg459-server [ 1661.329825] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:55:11 (1788814511) [ 1662.977472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg459-server [ 1664.529796] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:55:15 (1788814515) [ 1666.032542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg459-server [ 1667.429216] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:55:18 (1788814518) [ 1669.078785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg459-server [ 1671.062904] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:55:21 (1788814521) [ 1672.725934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg459-server [ 1674.689357] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:55:24 (1788814524) [ 1676.408326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg459-server [ 1677.879747] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:55:28 (1788814528) [ 1679.341308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg459-server [ 1681.015397] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:55:31 (1788814531) [ 1682.737396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg459-server [ 1684.314577] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:55:34 (1788814534) [ 1684.536878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.553462] LNet: 82783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.558892] LNetError: 82783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.567707] LNet: Removed LNI 192.168.204.59@tcp [ 1685.613296] LNet: Removed LNI 192.168.204.59@tcp1 [ 1685.640502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.723155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.748215] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1686.482481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1686.559773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1686.758409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.784367] LNet: 82930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.791131] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.799048] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.810394] LNet: Removed LNI 192.168.204.59@tcp [ 1686.842648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.897301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1686.919883] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1687.591773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1687.777798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.806474] LNetError: 83075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.870635] LNet: Removed LNI 192.168.204.59@tcp [ 1688.911288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.975549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.011709] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1689.547875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1689.623311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1689.774995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.803795] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.814180] LNet: 83222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1689.822377] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.840198] LNet: Removed LNI 192.168.204.59@tcp [ 1689.866761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.917712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1689.937324] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1690.570977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1690.740533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.814443] LNet: Removed LNI 192.168.204.59@tcp [ 1691.863316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.942604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.983148] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1692.730755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1692.842926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1693.111219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.182295] LNet: Removed LNI 192.168.204.59@tcp [ 1694.208764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.319053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1694.349372] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1694.999981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1695.230414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.258445] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.263872] LNet: 83660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1695.272678] LNetError: 83660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.278563] LNetError: 83660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1695.300154] LNet: Removed LNI 192.168.204.59@tcp [ 1695.327311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.394825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.426114] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1696.260681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1696.373815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1696.688126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.765987] LNet: Removed LNI 192.168.204.59@tcp [ 1697.800552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.881717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1697.911303] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1698.675363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1698.942506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.013113] LNet: Removed LNI 192.168.204.59@tcp [ 1700.058299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.127528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.165996] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1700.990448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1701.113892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1701.403518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.505960] LNet: Removed LNI 192.168.204.59@tcp [ 1702.541331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.607785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1702.633884] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1703.373839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1706.170458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.197977] LNet: 84265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.205770] LNet: 84265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1706.212186] LNetError: 84265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.218822] LNetError: 84265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1707.242845] LNet: Removed LNI 192.168.204.59@tcp [ 1707.277160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.356225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.385156] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1708.065560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1708.150180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1709.545419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.628896] LNet: Removed LNI 192.168.204.59@tcp [ 1710.660852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.714493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1710.739611] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1711.334310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1711.604718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.683808] LNet: Removed LNI 192.168.204.59@tcp [ 1712.708327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.764812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.785702] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1713.277252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1713.330645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.356427] LNet: Removed LNI 192.168.204.59@tcp [ 1713.374884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.419307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.437496] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1714.148080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1714.218327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.255966] LNet: Removed LNI 192.168.204.59@tcp [ 1714.281626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.333736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1714.412428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.448255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.485603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1714.561586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.588347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.628454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1714.700087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.725789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.761445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1714.776350] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1715.268816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.309506] LNet: Removed LNI 192.168.204.59@tcp [ 1715.335722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.409770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1721.233747] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:56:11 (1788814571) [ 1721.314763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.367233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.409606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1721.430396] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 1722.349756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.367503] LNet: 85726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.379678] LNet: 85726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1722.391598] LNetError: 85726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.401740] LNetError: 85726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1722.423301] LNet: Removed LNI 192.168.204.59@tcp231 [ 1722.435742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.487528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1722.507083] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 1722.511737] LNet: 85734:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.819214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1722.847869] LNet: Removed LNI 192.168.204.59@tcp231 [ 1722.862650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1722.888499] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 1723.961748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.998866] LNet: Removed LNI 192.168.204.59@tcp231 [ 1724.015213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.106738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1724.145805] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/0] [ 1724.155867] LNet: 85886:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1724.615838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1724.653917] LNet: Removed LNI 192.168.204.59@tcp231 [ 1724.675784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1724.707596] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/0] [ 1725.615975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.669547] LNet: Removed LNI 192.168.204.59@tcp231 [ 1725.693823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.753132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1725.776831] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/60] [ 1725.782414] LNet: 86037:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1726.053899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1726.078382] LNet: Removed LNI 192.168.204.59@tcp231 [ 1726.093988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1726.123814] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/60] [ 1727.325317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.362634] LNet: Removed LNI 192.168.204.59@tcp231 [ 1727.398631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.499816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1727.542153] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 1727.901643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.940307] LNet: Removed LNI 192.168.204.59@tcp231 [ 1727.964674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1727.997659] LNet: Added LNI 192.168.204.59@tcp231 [8/256/0/180] [ 1734.272959] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:56:25 (1788814585) [ 1734.388556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.437641] LNet: Removed LNI 192.168.204.59@tcp231 [ 1734.479673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.532971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1734.557628] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 1735.229330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1735.263179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1735.287729] LNet: Removed LNI 192.168.204.59@tcp232 [ 1735.301076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1735.319121] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 1735.968906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1736.019786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1736.056413] LNet: Removed LNI 192.168.204.59@tcp232 [ 1736.077733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1736.114331] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 1737.047665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1737.091612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1737.128064] LNet: Removed LNI 192.168.204.59@tcp232 [ 1737.160173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1737.205122] LNet: Added LNI 192.168.204.59@tcp232 [8/256/0/180] [ 1737.848420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1737.886508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1737.923589] LNet: Removed LNI 192.168.204.59@tcp232 [ 1737.942544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1743.493708] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:56:34 (1788814594) [ 1744.269326] Key type .llcrypt unregistered [ 1744.273314] Key type ._llcrypt unregistered [ 1749.299790] Key type ._llcrypt registered [ 1749.305375] Key type .llcrypt registered [ 1749.544575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1749.562154] alg: No test for adler32 (adler32-zlib) [ 1750.519858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1750.558308] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1757.730101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1764.508422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.541124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1764.560416] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1765.767461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_interrupt -r 2 [ 1765.815734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 2 [ 1765.854670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_interrupt -r 2 [ 1765.890867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 2 [ 1765.946688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_interrupt -r 2 [ 1765.982228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_interrupt -r 2 [ 1766.017547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_interrupt -r 2 [ 1766.055253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_interrupt -r 2 [ 1766.103798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1766.120749] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1766.127256] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1766.360931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.392321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.423736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.459400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.500052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1766.536038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.564357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.603303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.636841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.663953] LNet: 88687:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.690643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.726242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.758210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.796973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.821465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.860557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.248217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_dropped -r 2 [ 1767.284706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 2 [ 1767.321572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_dropped -r 2 [ 1767.352957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 2 [ 1767.379927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_dropped -r 2 [ 1767.408359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_dropped -r 2 [ 1767.441368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_dropped -r 2 [ 1767.467378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_dropped -r 2 [ 1767.502970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1767.532331] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1767.760617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.799382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.833605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1767.869569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1767.910894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1767.943212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.963951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.987879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.013330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.032797] LNet: 88810:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.054476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.088459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.112793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.139500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.183373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.234210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.652487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_aborted -r 2 [ 1768.684201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 2 [ 1768.722880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_aborted -r 2 [ 1768.766574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 2 [ 1768.807688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_aborted -r 2 [ 1768.842378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_aborted -r 2 [ 1768.882644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_aborted -r 2 [ 1768.920584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_aborted -r 2 [ 1768.951980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1768.976662] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1769.154428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.180559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.209766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.231901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.257716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1769.283652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.307798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.333623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.364561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.376679] LNet: 88932:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.395450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.428775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.459967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.494846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.526969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.561433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.939336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_no_route -r 2 [ 1769.966804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 2 [ 1770.009609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_no_route -r 2 [ 1770.038214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 2 [ 1770.067513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_no_route -r 2 [ 1770.099373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_no_route -r 2 [ 1770.129733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_no_route -r 2 [ 1770.159603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_no_route -r 2 [ 1770.199726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1770.423349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1770.454803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1770.481688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1770.508213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1770.542373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1770.566233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1770.600141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1770.638000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1770.659398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1770.689555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1770.711205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1770.743889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.773951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.803601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.834978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1771.279938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e local_timeout -r 2 [ 1771.321554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 2 [ 1771.377673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e local_timeout -r 2 [ 1771.419306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 2 [ 1771.482201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e local_timeout -r 2 [ 1771.519585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e local_timeout -r 2 [ 1771.556590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e local_timeout -r 2 [ 1771.594748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e local_timeout -r 2 [ 1771.622821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1771.640695] LNet: There was an unexpected network error while writing to 192.168.204.159: rc = -22 [ 1771.645272] LNet: Skipped 1 previous similar message [ 1771.870490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1771.904767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1771.939993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1771.974766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1772.001383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1772.027614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1772.054572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1772.089385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1772.125335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1772.148954] LNet: 89176:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.155656] LNet: 89176:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1772.176812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1772.222352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1772.250587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1772.288126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1772.319294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1772.355516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1772.783340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp -m GET -e remote_dropped -r 2 [ 1772.815640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 2 [ 1772.848181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp -m GET -e remote_dropped -r 2 [ 1772.883825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 2 [ 1772.918881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp -m GET -e remote_dropped -r 2 [ 1772.948548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.59@tcp1 -d 192.168.204.159@tcp1 -m GET -e remote_dropped -r 2 [ 1772.984765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp -m GET -e remote_dropped -r 2 [ 1773.015817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.59@tcp1 -d 192.168.204.59@tcp1 -m GET -e remote_dropped -r 2 [ 1773.050410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.59@tcp 192.168.204.159@tcp [ 1773.070657] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 1773.240810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1773.261652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1773.286319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1773.316869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1773.353144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1773.392811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1773.422478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1773.463290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1773.508479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1773.539188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1773.570307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1773.598313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1773.636168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1773.665396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1773.694365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1777.774127] LNet: 89373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.785344] LNetError: 89373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.792852] LNet: Removed LNI 192.168.204.59@tcp [ 1777.803432] LNet: Removed LNI 192.168.204.59@tcp1 [ 1778.129580] Key type .llcrypt unregistered [ 1778.134192] Key type ._llcrypt unregistered [ 1790.697430] Key type ._llcrypt registered [ 1790.700270] Key type .llcrypt registered [ 1791.823860] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:57:22 (1788814642) [ 1792.174377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.191480] alg: No test for adler32 (adler32-zlib) [ 1793.086449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.137267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.160635] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1793.797827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1793.826922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.59@tcp [ 1793.846373] LNetError: 90349:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 1793.903728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.59@tcp [ 1793.989880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.809348] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:57:29 (1788814649) [ 1798.875889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.890891] LNet: 90933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.898466] LNetError: 90933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.918074] LNet: Removed LNI 192.168.204.59@tcp [ 1798.935084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.979903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.998780] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1799.532166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1799.552759] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 1800.335118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.59@tcp [ 1800.353852] LNetError: 91205:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.59@tcp to recovery [ 1800.991448] LNet: 1 local NIs in recovery (showing 1): 192.168.204.59@tcp [ 1800.994357] LNet: There was an unexpected network error while writing to 192.168.204.59: rc = -22 [ 1801.571691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.59@tcp [ 1807.616885] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:57:38 (1788814658) [ 1809.118506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg459-server [ 1810.395229] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:57:41 (1788814661) [ 1811.814492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg459-server [ 1812.978851] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:57:43 (1788814663) [ 1814.291282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg459-server [ 1815.536489] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:57:46 (1788814666) [ 1815.815421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.833626] LNet: 92825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.838535] LNetError: 92825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.844576] LNet: Removed LNI 192.168.204.59@tcp [ 1815.854363] LNet: Removed LNI 192.168.204.59@tcp1 [ 1815.878526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.917230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1815.937564] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1816.504506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1816.570648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1816.611579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1816.710702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1816.750441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1816.805762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1821.473255] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:57:52 (1788814672) [ 1821.595937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.636085] LNet: 93556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.641600] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.647296] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1821.656176] LNet: Removed LNI 192.168.204.59@tcp [ 1821.671391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.722834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.746754] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1827.469781] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:57:58 (1788814678) [ 1827.578320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.602283] LNet: 94258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.609808] LNetError: 94258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.621595] LNet: Removed LNI 192.168.204.59@tcp [ 1827.636727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.679742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.701663] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1833.409986] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:58:04 (1788814684) [ 1834.533385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1835.898969] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:58:06 (1788814686) [ 1836.333888] LNet: 95294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.341607] LNetError: 95294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.355593] LNet: Removed LNI 192.168.204.59@tcp [ 1836.913206] Key type .llcrypt unregistered [ 1836.915535] Key type ._llcrypt unregistered [ 1841.798231] Key type ._llcrypt registered [ 1841.800734] Key type .llcrypt registered [ 1841.998846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1842.020103] alg: No test for adler32 (adler32-zlib) [ 1842.913169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1842.945772] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1848.822902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 1852.538363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.159@tcp [ 1852.856253] LNet: 95708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.864103] LNetError: 95708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.885292] LNet: Removed LNI 192.168.204.59@tcp [ 1853.249188] Key type .llcrypt unregistered [ 1853.252707] Key type ._llcrypt unregistered [ 1862.074313] Key type ._llcrypt registered [ 1862.076608] Key type .llcrypt registered [ 1863.054265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1864.240391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1865.418451] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:58:36 (1788814716) [ 1866.630480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1867.856677] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:58:38 (1788814718) [ 1869.225531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg459-server [ 1870.617388] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:58:41 (1788814721) [ 1872.086422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg459-server [ 1873.553447] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:58:44 (1788814724) [ 1874.914484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.951504] alg: No test for adler32 (adler32-zlib) [ 1875.978500] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1876.204103] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1877.863260] Key type lgssc registered [ 1878.861031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.574952] Key type lgssc unregistered [ 1894.816744] LNet: 98878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.825117] LNetError: 98878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.837939] LNet: Removed LNI 192.168.204.59@tcp [ 1895.420582] Key type .llcrypt unregistered [ 1895.425146] Key type ._llcrypt unregistered [ 1906.481829] Key type ._llcrypt registered [ 1906.485321] Key type .llcrypt registered [ 1907.591261] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:59:18 (1788814758) [ 1907.924762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.941490] alg: No test for adler32 (adler32-zlib) [ 1908.839405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.886781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.930514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.967257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1909.034714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1914.674925] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:59:25 (1788814765) [ 1914.819205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.843333] LNet: 100290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.863353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.902389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.932136] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1915.685912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.702644] LNet: 100434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.711625] LNetError: 100434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.727752] LNet: Removed LNI 192.168.204.59@tcp [ 1915.747956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.814198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1915.838862] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1916.262801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.282622] LNetError: 100463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.296926] LNet: Removed LNI 192.168.204.59@tcp [ 1916.312778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.361297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1916.385663] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1922.091510] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:59:32 (1788814772) [ 1922.189782] LNet: 101051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.194800] LNet: 101051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1922.199715] LNetError: 101051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.222967] LNet: Removed LNI 192.168.204.59@tcp [ 1922.769580] Key type .llcrypt unregistered [ 1922.778419] Key type ._llcrypt unregistered [ 1928.176400] Key type ._llcrypt registered [ 1928.180365] Key type .llcrypt registered [ 1928.373464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1928.393431] alg: No test for adler32 (adler32-zlib) [ 1929.310267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.391788] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1929.395103] LNet: *** cfs_fail_loc=e003, val=100*** [ 1930.092375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1930.108852] LNet: *** cfs_fail_loc=e003, val=100*** [ 1930.111999] LNet: Skipped 2 previous similar messages [ 1930.118443] Ping reply received with mapping: [ 1930.124134] found metadata reply NI[192.168.204.59@tcp] [ 1930.127180] 3b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1930.129336] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1930.133197] 5a 5a 5a 5a 5a 5a 5a 5a [ 1930.247845] LNet: 101447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.251761] LNetError: 101447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.308621] LNet: Removed LNI 192.168.204.59@tcp [ 1931.908511] Key type .llcrypt unregistered [ 1931.910554] Key type ._llcrypt unregistered [ 1940.540595] Key type ._llcrypt registered [ 1940.543464] Key type .llcrypt registered [ 1941.763896] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:59:52 (1788814792) [ 1942.181382] Key type .llcrypt unregistered [ 1942.191178] Key type ._llcrypt unregistered [ 1947.038250] Key type ._llcrypt registered [ 1947.042172] Key type .llcrypt registered [ 1947.239976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1947.250935] alg: No test for adler32 (adler32-zlib) [ 1948.314991] LNetError: 102483:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1948.668577] Key type .llcrypt unregistered [ 1948.670972] Key type ._llcrypt unregistered [ 1952.622721] Key type ._llcrypt registered [ 1952.624464] Key type .llcrypt registered [ 1953.715326] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:00:04 (1788814804) [ 1954.057491] Key type .llcrypt unregistered [ 1954.062587] Key type ._llcrypt unregistered [ 1958.497605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1958.502775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1960.905715] Key type ._llcrypt registered [ 1960.907747] Key type .llcrypt registered [ 1961.071231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.082654] alg: No test for adler32 (adler32-zlib) [ 1961.958409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.015749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1969.603067] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:00:20 (1788814820) [ 1969.679373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.702064] LNet: 104249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.709771] LNetError: 104249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1969.729091] LNet: Removed LNI 10.1.2.3@tcp [ 1969.755774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.375677] Key type .llcrypt unregistered [ 1970.377642] Key type ._llcrypt unregistered [ 1975.017826] Key type ._llcrypt registered [ 1975.021178] Key type .llcrypt registered [ 1975.166811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1975.184619] alg: No test for adler32 (adler32-zlib) [ 1976.134491] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 1976.192927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 1976.322286] LNet: 104542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.328360] LNetError: 104542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.345284] LNet: Removed LNI 192.168.204.59@tcp [ 1976.884435] Key type .llcrypt unregistered [ 1976.887725] Key type ._llcrypt unregistered [ 1986.067930] Key type ._llcrypt registered [ 1986.070915] Key type .llcrypt registered [ 1987.113399] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:00:37 (1788814837) [ 1987.764404] Key type .llcrypt unregistered [ 1987.767460] Key type ._llcrypt unregistered [ 1993.285227] Key type ._llcrypt registered [ 1993.293128] Key type .llcrypt registered [ 1993.513965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1993.536903] alg: No test for adler32 (adler32-zlib) [ 2000.332966] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:00:50 (1788814850) [ 2000.438486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.469889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.537789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.567354] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2001.157880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.206045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2001.229920] LNet: Removed LNI 192.168.204.59@tcp [ 2001.417933] LNet: 106282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.756827] Key type .llcrypt unregistered [ 2001.759465] Key type ._llcrypt unregistered [ 2011.619287] Key type ._llcrypt registered [ 2011.621687] Key type .llcrypt registered [ 2012.747607] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:01:03 (1788814863) [ 2013.189613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2013.216920] alg: No test for adler32 (adler32-zlib) [ 2014.226496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.285435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2014.305834] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2015.058773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2015.100812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.59@tcp [ 2015.121129] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2015.125675] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.130113] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 0 [ 2015.137728] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2015.140460] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2015.143784] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.146755] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 0 [ 2015.150100] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2015.152957] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2015.156405] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.159161] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 2065 [ 2015.163466] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2015.165945] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2015.169377] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.172499] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 2065 [ 2015.175938] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2020.665372] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:01:11 (1788814871) [ 2021.022947] LNet: 107833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.030428] LNetError: 107833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.043648] LNet: Removed LNI 192.168.204.59@tcp [ 2021.723398] Key type .llcrypt unregistered [ 2021.726665] Key type ._llcrypt unregistered [ 2027.522508] Key type ._llcrypt registered [ 2027.526259] Key type .llcrypt registered [ 2027.700158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2027.716517] alg: No test for adler32 (adler32-zlib) [ 2028.589826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2028.609446] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2034.954693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 2043.399918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.459610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2043.503866] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 2044.370686] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2044.594184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2044.597576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2044.608690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2046.883609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2046.911423] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2056.180318] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:01:46 (1788814906) [ 2056.575203] LNet: 109339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.582784] LNetError: 109339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.587916] LNet: Removed LNI 192.168.204.59@tcp [ 2057.631181] LNet: Removed LNI 192.168.204.59@tcp1 [ 2057.640237] LNetError: 109339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2057.648922] LNetError: 109339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2057.678233] LNet: Removed LNI 10.1.2.3@tcp99 [ 2058.289888] Key type .llcrypt unregistered [ 2058.295369] Key type ._llcrypt unregistered [ 2064.263340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2064.265988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2066.722922] Key type ._llcrypt registered [ 2066.726875] Key type .llcrypt registered [ 2066.909937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2066.918932] alg: No test for adler32 (adler32-zlib) [ 2067.829918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.877446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.901827] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2068.399238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2068.431461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2070.211697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2070.263296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2070.304585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2070.330639] LNetError: 110081:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2070.371038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2070.432300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2070.794584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2070.849180] LNet: 110101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.854185] LNetError: 110101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2070.858886] LNet: Removed LNI 10.1.2.3@tcp [ 2071.910676] LNet: Removed LNI 192.168.204.59@tcp [ 2072.652597] Key type .llcrypt unregistered [ 2072.655540] Key type ._llcrypt unregistered [ 2081.766879] Key type ._llcrypt registered [ 2081.769214] Key type .llcrypt registered [ 2082.890586] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:02:13 (1788814933) [ 2083.166174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.185711] alg: No test for adler32 (adler32-zlib) [ 2084.103803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.188072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.208234] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2084.830648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2084.945086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg459-client@tcp [ 2090.613408] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:02:21 (1788814941) [ 2090.691226] LNet: 111623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.716872] LNetError: 111623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.747362] LNet: Removed LNI 192.168.204.59@tcp [ 2091.245492] Key type .llcrypt unregistered [ 2091.247723] Key type ._llcrypt unregistered [ 2096.081914] Key type ._llcrypt registered [ 2096.083936] Key type .llcrypt registered [ 2096.205649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.217602] alg: No test for adler32 (adler32-zlib) [ 2097.123326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2097.149994] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2097.180367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2102.052927] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:02:32 (1788814952) [ 2103.255915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2104.754799] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:02:35 (1788814955) [ 2106.140893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2107.565651] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:02:38 (1788814958) [ 2107.632765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.655591] LNet: 113042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.658868] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.677748] LNet: Removed LNI 192.168.204.59@tcp [ 2107.697287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.766143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.790286] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2108.445281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2108.494951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2113.505391] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:02:44 (1788814964) [ 2113.646882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.663496] LNet: 113755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.670770] LNetError: 113755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.725765] LNet: Removed LNI 192.168.204.59@tcp [ 2114.765353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.853562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2114.889945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2115.100863] LNet: 113795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.404504] Key type .llcrypt unregistered [ 2115.409282] Key type ._llcrypt unregistered [ 2124.717542] Key type ._llcrypt registered [ 2124.719369] Key type .llcrypt registered [ 2125.812284] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:02:56 (1788814976) [ 2126.230149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.243617] alg: No test for adler32 (adler32-zlib) [ 2127.165874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.244606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.271629] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2127.888357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2127.932970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2134.062659] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:03:04 (1788814984) [ 2134.152580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.180639] LNet: 115305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.188779] LNetError: 115305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.220052] LNet: Removed LNI 192.168.204.59@tcp [ 2134.261474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.334208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2134.371858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2139.875982] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:03:10 (1788814990) [ 2139.948240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.961367] LNet: 115885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.978271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.034774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2140.292810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2140.330817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2140.404827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2140.739361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2140.793815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2140.854273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.885680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.950204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2141.249788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2141.298712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2141.371257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2141.780668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2141.818791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2146.720706] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:03:17 (1788814997) [ 2146.911982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.925860] LNet: 116525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.930961] LNet: 116525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2146.949509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.995088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.020795] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2147.593746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2147.613589] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2148.325573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2148.365689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2148.563972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.587695] LNetError: 116804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.595419] LNet: Removed LNI 192.168.204.59@tcp [ 2148.606448] LNet: Removed LNI 192.168.204.59@tcp2 [ 2148.625808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.677993] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2148.683570] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2148.687985] LNet: 116810:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.940876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2149.089684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.113335] LNet: 116847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.122492] LNet: 116847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2150.175120] LNet: Removed LNI 192.168.204.59@tcp [ 2150.176978] LNetError: 116847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.182639] LNetError: 116847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2150.194334] LNet: Removed LNI 192.168.204.59@tcp2 [ 2150.235131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.311413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.340801] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2150.894747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.913430] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2151.671459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2151.702429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2151.753920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2151.928979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.991205] LNet: Removed LNI 192.168.204.59@tcp [ 2153.007329] LNetError: 117131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.016978] LNetError: 117131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2153.034474] LNet: Removed LNI 192.168.204.59@tcp2 [ 2153.049292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.096425] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2153.101731] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2153.106081] LNet: 117138:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.490907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2153.674250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.696229] LNet: 117175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.700519] LNet: 117175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2153.706733] LNet: Removed LNI 192.168.204.59@tcp [ 2154.727736] LNet: Removed LNI 192.168.204.59@tcp2 [ 2154.747177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.804623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.829043] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2155.317761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2155.342374] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2156.083445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2156.150042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2156.318222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.343347] LNet: Removed LNI 192.168.204.59@tcp [ 2157.346157] LNetError: 117454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.351132] LNetError: 117454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2157.361940] LNet: Removed LNI 192.168.204.59@tcp2 [ 2157.392696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.483037] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2157.492170] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2157.499026] LNet: 117460:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2157.976379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2158.171949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.199255] LNet: Removed LNI 192.168.204.59@tcp [ 2159.216647] LNet: Removed LNI 192.168.204.59@tcp2 [ 2159.275732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.364091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.399146] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2160.099463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2160.122533] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2160.803608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2160.841739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2160.889615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2161.064467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.143182] LNet: Removed LNI 192.168.204.59@tcp [ 2163.175517] LNet: Removed LNI 192.168.204.59@tcp2 [ 2163.213944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.270517] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2163.278563] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2163.281552] LNet: 117789:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2163.759686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2163.974082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.994285] LNet: 117826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.001638] LNet: 117826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2164.012794] LNet: Removed LNI 192.168.204.59@tcp [ 2165.029694] LNet: Removed LNI 192.168.204.59@tcp2 [ 2165.066461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.129343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.164339] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2165.688447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2165.704931] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2166.405305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2166.425371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2166.473703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2166.772842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2166.790793] LNetError: 118122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.794702] LNetError: 118122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2166.803786] LNet: Removed LNI 192.168.204.59@tcp [ 2166.815529] LNet: Removed LNI 192.168.204.59@tcp2 [ 2166.841148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.890049] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2166.895749] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2167.322931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2167.653542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.735574] LNet: Removed LNI 192.168.204.59@tcp [ 2168.755546] LNet: Removed LNI 192.168.204.59@tcp2 [ 2168.812669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.919658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.961731] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2169.570906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2169.601583] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2170.278820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2170.310376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2170.333783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2170.378454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2170.674714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.743178] LNet: Removed LNI 192.168.204.59@tcp [ 2172.775373] LNet: Removed LNI 192.168.204.59@tcp2 [ 2172.821215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.884760] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2172.892499] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2172.898368] LNet: 118488:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2172.903979] LNet: 118488:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2173.419732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2179.755367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2181.398179] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:03:52 (1788815032) [ 2181.563941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.581128] LNet: 119251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.588510] LNet: 119251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2181.594573] LNet: Removed LNI 192.168.204.59@tcp [ 2182.640950] LNet: Removed LNI 192.168.204.59@tcp2 [ 2182.678749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.956211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.990057] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2183.633204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2183.655331] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.407491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.434072] LNetError: 119526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2184.443680] LNetError: 119526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2184.451691] LNet: Removed LNI 10.1.2.3@tcp [ 2184.478568] LNet: Removed LNI 192.168.204.59@tcp [ 2184.503263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.597053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2184.626867] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2184.634710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2185.017578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2185.038493] LNet: Removed LNI 192.168.204.59@tcp [ 2185.059131] LNet: Removed LNI 10.1.2.3@tcp [ 2190.167455] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:04:00 (1788815040) [ 2190.301052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.350591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.383359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.407148] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2190.946638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2190.975401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2191.009425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.046294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.079861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.121812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.148184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.191685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.227511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.264813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.297919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.333340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2191.497386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2196.258665] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 17:04:06 (1788815046) [ 2196.377460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.428495] LNet: Removed LNI 192.168.204.59@tcp [ 2197.456231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.510451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.555430] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2198.319579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2198.364233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2198.416412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.459465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.499879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.529854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.557675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.605390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.637493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.663175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.692287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.719710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.788465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2198.823536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.852280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.881233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.906898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.932485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.956498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2198.986873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2199.011622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2199.038462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2199.084401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.59@tcp [ 2204.357118] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:04:15 (1788815055) [ 2204.417692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.484389] LNet: Removed LNI 192.168.204.59@tcp [ 2205.525077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.680540] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2205.852715] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2205.857213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2205.859801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2208.103537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2208.130922] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2208.751774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2208.772892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2210.384249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2210.433639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2210.486173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.59@tcp -m PUT -e local_timeout -r 1 [ 2210.560407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2210.637128] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.644474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2210.820643] LNetError: 121692:0:(lib-move.c:4794:lnet_parse()) 192.168.204.59@tcp, src 192.168.204.59@tcp: Dropping PUT (error -108 looking up sender) [ 2211.876319] LNet: Removed LNI 10.1.2.3@tcp [ 2212.906630] LNet: Removed LNI 192.168.204.59@tcp [ 2213.251431] Key type .llcrypt unregistered [ 2213.254101] Key type ._llcrypt unregistered [ 2222.462285] Key type ._llcrypt registered [ 2222.465095] Key type .llcrypt registered [ 2223.472607] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:04:34 (1788815074) [ 2223.831775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.847498] alg: No test for adler32 (adler32-zlib) [ 2224.774117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.108353] LNet: 122999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.705491] Key type .llcrypt unregistered [ 2225.707877] Key type ._llcrypt unregistered [ 2230.728599] Key type ._llcrypt registered [ 2230.732654] Key type .llcrypt registered [ 2230.905727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2230.921470] alg: No test for adler32 (adler32-zlib) [ 2231.820029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2231.847992] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2238.543295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 2238.717278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2239.366368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.159@tcp [ 2239.635794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2239.668588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2239.711141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2239.741411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2239.776188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2239.806675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2239.833300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2239.869371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.902263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.930504] LNet: 123448:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.951850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2239.992951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2240.035091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2240.036474] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 2240.071267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2240.124200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2240.174458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2240.741110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2240.938043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2240.972324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2241.005181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2241.051808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2241.080585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2241.108285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2241.132105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2241.156861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2241.185242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2241.204696] LNet: 123567:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2241.220166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2241.247876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2241.273586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2241.304760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2241.330134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2241.365814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.891938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.159@tcp [ 2242.144421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2242.180367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2242.204861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.241720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.278376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2242.306405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.337274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.365261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.397537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.413221] LNet: 123686:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.431975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.458772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.489950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2242.529628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2242.561536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2242.588978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.096054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2243.306557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2243.327603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2243.349361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2243.370949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2243.410123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2243.437932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2243.469591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2243.495130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2243.513805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2243.534947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2243.554643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2243.583302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2243.611951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2243.657696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.690969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.934263] LNet: 123851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.941104] LNetError: 123851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.960422] LNet: Removed LNI 192.168.204.59@tcp [ 2244.290958] Key type .llcrypt unregistered [ 2244.294109] Key type ._llcrypt unregistered [ 2257.157886] Key type ._llcrypt registered [ 2257.161520] Key type .llcrypt registered [ 2258.197071] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:05:08 (1788815108) [ 2258.553821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.576757] alg: No test for adler32 (adler32-zlib) [ 2259.486334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.844122] LNet: 124735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.338426] Key type .llcrypt unregistered [ 2260.343649] Key type ._llcrypt unregistered [ 2265.314163] Key type ._llcrypt registered [ 2265.318849] Key type .llcrypt registered [ 2265.487830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2265.501201] alg: No test for adler32 (adler32-zlib) [ 2266.400252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2266.424460] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2272.728669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.159@tcp [ 2280.662486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.724979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2280.763577] LNet: Added LNI 192.168.204.59@tcp1 [8/256/0/180] [ 2281.810278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2282.465822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.159@tcp [ 2282.703503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2282.741801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2282.777830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2282.809106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2282.836613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2282.850568] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.159@tcp [ 2282.882731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2282.917582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2282.944851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2282.980671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2283.002732] LNet: 125385:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.029359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2283.078410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2283.116860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2283.171397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2283.203508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2283.236372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2283.813098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2284.017666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.047140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2284.084160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2284.126505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2284.156162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2284.180334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2284.208382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2284.233955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2284.256879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2284.272649] LNet: 125504:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2284.287370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2284.311325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2284.341951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2284.383226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2284.422952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2284.446907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2285.002458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.159@tcp [ 2285.208623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2285.234876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2285.257568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2285.285210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2285.311740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2285.344950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2285.382595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2285.434925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2285.474653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2285.495636] LNet: 125623:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.514368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2285.548298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2285.585948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2285.636449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2285.673559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.706702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.375400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2286.611697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.636478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.666536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.698649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.730855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2286.761910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.786276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.825968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.869291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.895751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.927894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.977970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2287.008471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2287.036693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.072406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2291.163374] LNet: 125823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.167793] LNetError: 125823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.173616] LNet: Removed LNI 192.168.204.59@tcp [ 2291.189515] LNet: Removed LNI 192.168.204.59@tcp1 [ 2291.486341] Key type .llcrypt unregistered [ 2291.488064] Key type ._llcrypt unregistered [ 2305.104896] Key type ._llcrypt registered [ 2305.110208] Key type .llcrypt registered [ 2305.997775] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:05:56 (1788815156) [ 2306.412894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.429563] alg: No test for adler32 (adler32-zlib) [ 2307.311922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.366798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.384162] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2307.945771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2307.962208] LNet: Added LNI 192.168.204.59@tcp2 [8/256/0/180] [ 2308.742360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.59@tcp [ 2308.803516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2309.855247] LNet: Removed LNI 192.168.204.59@tcp [ 2314.963179] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:06:05 (1788815165) [ 2316.244623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2317.725769] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:06:08 (1788815168) [ 2323.131767] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:06:13 (1788815173) [ 2328.586957] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:06:19 (1788815179) [ 2333.900626] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:06:24 (1788815184) [ 2339.306417] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:06:30 (1788815190) [ 2344.687489] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:06:35 (1788815195) [ 2349.614617] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:06:40 (1788815200) [ 2354.474309] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:06:45 (1788815205) [ 2359.906802] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:06:50 (1788815210) [ 2364.719885] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:06:55 (1788815215) [ 2370.125310] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:07:00 (1788815220) [ 2375.790074] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:07:06 (1788815226) [ 2380.898511] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:07:11 (1788815231) [ 2386.165703] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:07:16 (1788815236) [ 2391.619599] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:07:22 (1788815242) [ 2396.688377] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:07:27 (1788815247) [ 2401.417537] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:07:32 (1788815252) [ 2406.433173] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:07:37 (1788815257) [ 2406.570325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.582342] LNet: 137509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.589413] LNetError: 137509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.601910] LNet: Removed LNI 192.168.204.59@tcp2 [ 2406.618227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.700838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.730935] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2407.455387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2407.468176] LNet: 137651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.472650] LNetError: 137651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2407.482589] LNet: Removed LNI 192.168.204.59@tcp [ 2407.502547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.566459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2407.599682] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2412.595194] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:07:43 (1788815263) [ 2418.273714] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:07:48 (1788815268) [ 2423.047501] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:07:53 (1788815273) [ 2427.854410] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:07:58 (1788815278) [ 2432.732923] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:08:03 (1788815283) [ 2438.200916] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:08:08 (1788815288) [ 2443.776432] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:08:14 (1788815294) [ 2448.940471] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:08:19 (1788815299) [ 2453.685681] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:08:24 (1788815304) [ 2458.731577] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:08:29 (1788815309) [ 2463.777750] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:08:34 (1788815314) [ 2468.513972] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:08:39 (1788815319) [ 2473.018020] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2245 sec ========== 17:08:43 (1788815323) [ 2473.204487] LNet: 145114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.208774] LNetError: 145114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.223547] LNet: Removed LNI 192.168.204.59@tcp [ 2473.760921] Key type .llcrypt unregistered [ 2473.764866] Key type ._llcrypt unregistered [ 2483.147263] Key type ._llcrypt registered [ 2483.149030] Key type .llcrypt registered [ 2483.421061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2483.430824] alg: No test for adler32 (adler32-zlib) [ 2484.464776] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2484.660418] LNet: Added LNI 192.168.204.59@tcp [8/256/0/180] [ 2486.311324] Key type lgssc registered [ 2487.203507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2535.755930] Lustre: Mounted lustre-client - version 2.17.58_41_g50dbf09 [ 2539.886829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2545.716101] Lustre: Unmounted lustre-client [ 2566.154088] Key type lgssc unregistered [ 2566.420579] LNet: 148058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.426346] LNetError: 148058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.446493] LNet: Removed LNI 192.168.204.59@tcp [ 2567.067267] Key type .llcrypt unregistered [ 2567.072794] Key type ._llcrypt unregistered