[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 591555572 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003254] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.011000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012013] pid_max: default: 32768 minimum: 301 [ 0.013132] LSM: Security Framework initializing [ 0.014066] Yama: becoming mindful. [ 0.015058] SELinux: Initializing. [ 0.016096] *** VALIDATE selinux *** [ 0.026608] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031562] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034111] *** VALIDATE tmpfs *** [ 0.035457] *** VALIDATE proc *** [ 0.036265] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.039256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042034] Spectre V2 : User space: Vulnerable [ 0.043013] Speculative Store Bypass: Vulnerable [ 0.046111] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.048244] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049750] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051156] ... version: 2 [ 0.052129] ... bit width: 48 [ 0.053017] ... generic registers: 4 [ 0.054016] ... value mask: 0000ffffffffffff [ 0.055018] ... max period: 00007fffffffffff [ 0.056017] ... fixed-purpose events: 3 [ 0.057118] ... event mask: 000000070000000f [ 0.058456] rcu: Hierarchical SRCU implementation. [ 0.060763] smp: Bringing up secondary CPUs ... [ 0.061751] x86: Booting SMP configuration: [ 0.062031] .... node #0, CPUs: #1 #2 #3 [ 0.071202] smp: Brought up 1 node, 4 CPUs [ 0.073030] smpboot: Max logical packages: 1 [ 0.074060] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.118022] node 0 deferred pages initialised in 42ms [ 0.132256] devtmpfs: initialized [ 0.135000] x86/mm: Memory block size: 128MB [ 0.140391] gcov: version magic: 0x41383552 [ 0.150620] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168265] pinctrl core: initialized pinctrl subsystem [ 0.174177] [ 0.174645] ************************************************************* [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184023] ** ** [ 0.191023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.197013] ** ** [ 0.210013] ** This means that this kernel is built to expose internal ** [ 0.220015] ** IOMMU data structures, which may compromise security on ** [ 0.227015] ** your system. ** [ 0.237012] ** ** [ 0.248021] ** If you see this message and you are not debugging the ** [ 0.251017] ** kernel, report this immediately to your vendor! ** [ 0.252016] ** ** [ 0.253000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253000] ************************************************************* [ 0.254856] NET: Registered protocol family 16 [ 0.257473] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260144] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267011] cpuidle: using governor menu [ 0.270013] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273539] PCI: Using configuration type 1 for base access [ 0.276129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.290184] cryptd: max_cpu_qlen set to 1000 [ 0.291000] ACPI: Added _OSI(Module Device) [ 0.291014] ACPI: Added _OSI(Processor Device) [ 0.292000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.294014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299916] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306256] ACPI: Interpreter enabled [ 0.308101] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311013] ACPI: Using IOAPIC for interrupt routing [ 0.313094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317518] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.333000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.335042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349729] acpiphp: Slot [2] registered [ 0.352311] acpiphp: Slot [5] registered [ 0.354273] acpiphp: Slot [6] registered [ 0.356124] acpiphp: Slot [3] registered [ 0.358107] acpiphp: Slot [4] registered [ 0.360109] acpiphp: Slot [7] registered [ 0.362134] acpiphp: Slot [8] registered [ 0.364113] acpiphp: Slot [9] registered [ 0.366400] acpiphp: Slot [10] registered [ 0.368176] acpiphp: Slot [11] registered [ 0.371111] acpiphp: Slot [12] registered [ 0.374116] acpiphp: Slot [13] registered [ 0.376336] acpiphp: Slot [14] registered [ 0.379116] acpiphp: Slot [15] registered [ 0.381202] acpiphp: Slot [16] registered [ 0.384129] acpiphp: Slot [17] registered [ 0.385211] acpiphp: Slot [18] registered [ 0.386311] acpiphp: Slot [19] registered [ 0.387000] acpiphp: Slot [20] registered [ 0.387000] acpiphp: Slot [21] registered [ 0.387000] acpiphp: Slot [22] registered [ 0.387000] acpiphp: Slot [23] registered [ 0.387000] acpiphp: Slot [24] registered [ 0.387000] acpiphp: Slot [25] registered [ 0.387000] acpiphp: Slot [26] registered [ 0.387000] acpiphp: Slot [27] registered [ 0.387000] acpiphp: Slot [28] registered [ 0.387000] acpiphp: Slot [29] registered [ 0.387000] acpiphp: Slot [30] registered [ 0.387000] acpiphp: Slot [31] registered [ 0.387000] PCI host bridge to bus 0000:00 [ 0.387000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.387000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.387000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.387000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.387000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.387000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.387000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.387000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.396014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.399000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.399000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.399000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.410125] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.413924] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.415061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.425000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437208] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458199] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522315] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.529293] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.533607] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.537579] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.542000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.547012] iommu: Default domain type: Passthrough [ 0.551077] SCSI subsystem initialized [ 0.553164] ACPI: bus type USB registered [ 0.556164] usbcore: registered new interface driver usbfs [ 0.559084] usbcore: registered new interface driver hub [ 0.562110] usbcore: registered new device driver usb [ 0.564191] pps_core: LinuxPPS API ver. 1 registered [ 0.567013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.572057] PTP clock support registered [ 0.575652] EDAC MC: Ver: 3.0.0 [ 0.578133] PCI: Using ACPI for IRQ routing [ 0.579950] NetLabel: Initializing [ 0.582013] NetLabel: domain hash size = 128 [ 0.584011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.586243] NetLabel: unlabeled traffic allowed by default [ 0.590275] vgaarb: loaded [ 0.592410] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.604268] clocksource: Switched to clocksource kvm-clock [ 0.756581] VFS: Disk quotas dquot_6.6.0 [ 0.758715] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.762290] *** VALIDATE ramfs *** [ 0.763880] *** VALIDATE hugetlbfs *** [ 0.765972] pnp: PnP ACPI init [ 0.768949] pnp: PnP ACPI: found 6 devices [ 0.785573] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.789975] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.792969] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795945] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.799341] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.802693] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.806513] NET: Registered protocol family 2 [ 0.809587] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.815698] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.820418] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.830379] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.837633] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843744] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.850503] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856859] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864411] NET: Registered protocol family 1 [ 0.870064] RPC: Registered named UNIX socket transport module. [ 0.878411] RPC: Registered udp transport module. [ 0.880770] RPC: Registered tcp transport module. [ 0.884948] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.888365] NET: Registered protocol family 44 [ 0.891890] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.897274] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901245] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.906427] PCI: CLS 0 bytes, default 64 [ 0.912036] Unpacking initramfs... [ 3.616708] debug: unmapping init [mem 0xffff9e77bcc64000-0xffff9e77bffcffff] [ 3.620944] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.623287] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 3.626955] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.890205] Initialise system trusted keyrings [ 4.892360] Key type blacklist registered [ 4.894853] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.907840] zbud: loaded [ 4.911211] *** VALIDATE nfs *** [ 4.912718] *** VALIDATE nfs4 *** [ 4.915136] pstore: using deflate compression [ 4.921287] Platform Keyring initialized [ 5.147948] NET: Registered protocol family 38 [ 5.154210] Key type asymmetric registered [ 5.157730] Asymmetric key parser 'x509' registered [ 5.160350] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.166666] io scheduler mq-deadline registered [ 5.169083] io scheduler kyber registered [ 5.171449] io scheduler bfq registered [ 5.174205] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.178385] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.182419] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.186254] ACPI: Power Button [PWRF] [ 5.192874] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.200526] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.230105] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.259361] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.292166] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.307603] Non-volatile memory driver v1.3 [ 5.310293] Linux agpgart interface v0.103 [ 5.436498] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.441540] vda: detected capacity change from 0 to 74698752 [ 5.475832] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.489666] vdb: detected capacity change from 0 to 1073741824 [ 5.626741] libphy: Fixed MDIO Bus: probed [ 5.632941] usbcore: registered new interface driver usbserial_generic [ 5.635757] usbserial: USB Serial support registered for generic [ 5.638350] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.643093] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.645287] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.647806] mousedev: PS/2 mouse device common for all mice [ 5.651916] rtc_cmos 00:05: RTC can wake from S4 [ 5.659467] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.662943] rtc_cmos 00:05: registered as rtc0 [ 5.676785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.680367] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.680437] intel_pstate: CPU model not supported [ 5.684502] hid: raw HID events driver (C) Jiri Kosina [ 5.689097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.750532] usbcore: registered new interface driver usbhid [ 5.750540] usbhid: USB HID core driver [ 5.750693] drop_monitor: Initializing network drop monitor service [ 5.753039] Initializing XFRM netlink socket [ 5.775276] NET: Registered protocol family 10 [ 5.779192] Segment Routing with IPv6 [ 5.781139] NET: Registered protocol family 17 [ 5.784808] mpls_gso: MPLS GSO support [ 5.793542] RAS: Correctable Errors collector initialized. [ 5.803559] AVX version of gcm_enc/dec engaged. [ 5.808921] AES CTR mode by8 optimization enabled [ 6.085925] sched_clock: Marking stable (6085832922, 0)->(7647801443, -1561968521) [ 6.090725] registered taskstats version 1 [ 6.093382] Loading compiled-in X.509 certificates [ 6.096121] zswap: loaded using pool lzo/zbud [ 6.127488] Key type big_key registered [ 6.144139] Key type encrypted registered [ 6.146330] ima: No TPM chip found, activating TPM-bypass! [ 6.151413] ima: Allocated hash algorithm: sha1 [ 6.154893] ima: No architecture policies found [ 6.162424] evm: Initialising EVM extended attributes: [ 6.168689] evm: security.selinux [ 6.171405] evm: security.ima [ 6.172887] evm: security.capability [ 6.174823] evm: HMAC attrs: 0x1 [ 6.178572] rtc_cmos 00:05: setting system clock to 2026-08-08 09:55:32 UTC (1786182932) [ 6.187800] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 6.194027] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 6.203196] Write protecting the kernel read-only data: 28672k [ 6.207482] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 6.210832] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 6.259350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.269849] systemd[1]: Detected virtualization kvm. [ 6.272352] systemd[1]: Detected architecture x86-64. [ 6.274872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.304699] systemd[1]: No hostname configured. [ 6.306828] systemd[1]: Set hostname to . [ 6.309165] random: systemd: uninitialized urandom read (16 bytes read) [ 6.312185] systemd[1]: Initializing machine ID from random generator. [ 6.400735] random: ln: uninitialized urandom read (6 bytes read) [ 6.583790] random: systemd: uninitialized urandom read (16 bytes read) [ 6.587690] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.612500] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.636118] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.722659] device-mapper: uevent: version 1.0.3 [ 8.725452] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.619219] virtio_net virtio0 ens2: renamed from eth0 [ 11.078209] scsi host0: ata_piix [ 11.095349] scsi host1: ata_piix [ 11.101265] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.110731] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.378037] random: fast init done [ 16.787523] random: crng init done [ 16.790719] random: 7 urandom warning(s) missed due to ratelimiting [ 19.247875] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.349547] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.286331] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.863596] SELinux: Disabled at runtime. [ 24.932522] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.943445] systemd[1]: Detected virtualization kvm. [ 24.945702] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.152162] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.162942] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.177421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.182313] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.186471] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.198165] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.211704] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 27.559214] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.224273] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.217124] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.525199] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.602960] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.997468] EDAC sbridge: Ver: 1.1.2 [ 34.740490] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (9s / no limit)[ 35.634417] NFS: Registering the id_resolver key type [ 35.651985] Key type id_resolver registered [ 35.673168] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg331-client login: [ 93.189851] libcfs: loading out-of-tree module taints kernel. [ 93.295900] Key type ._llcrypt registered [ 93.302770] Key type .llcrypt registered [ 93.844404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 93.855058] alg: No test for adler32 (adler32-zlib) [ 95.155943] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 95.693284] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 97.516517] Key type lgssc registered [ 99.534108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 217.736544] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc [ 222.588294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.489457] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing check_logdir /tmp/testlogs/ [ 243.330936] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing yml_node [ 243.681285] Lustre: lustre-OST0000-osc-ffff9e7804c5e800: disconnect after 24s idle [ 247.124864] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 249.712977] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 252.166401] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 254.396394] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 8 05:59:38 EDT 2026 [ 268.651969] Lustre: Unmounted lustre-client [ 305.252922] Key type lgssc unregistered [ 305.493283] LNet: 8437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.500776] LNetError: 8437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.545343] LNet: Removed LNI 192.168.203.31@tcp [ 307.073236] Key type .llcrypt unregistered [ 307.075285] Key type ._llcrypt unregistered [ 330.711530] Key type ._llcrypt registered [ 330.714555] Key type .llcrypt registered [ 330.904728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 330.916287] alg: No test for adler32 (adler32-zlib) [ 332.166642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 332.224058] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 332.241573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 334.926228] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 335.281565] LNet: 9329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.348681] LNetError: 9329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.366385] LNet: Removed LNI 192.168.203.31@tcp [ 335.963589] Key type .llcrypt unregistered [ 335.967257] Key type ._llcrypt unregistered [ 341.277705] Key type ._llcrypt registered [ 341.280441] Key type .llcrypt registered [ 342.212561] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:01:07 (1786183267) [ 343.051429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.061579] alg: No test for adler32 (adler32-zlib) [ 343.999471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.139545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.158803] LNet: 9754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.175361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.241466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.273314] LNet: 9761:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 350.981193] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:01:15 (1786183275) [ 351.267096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.288763] LNet: 10341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.313685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.421504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 357.603348] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:01:22 (1786183282) [ 357.950808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.974954] LNet: 10926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.987683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.046936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.910204] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:01:29 (1786183289) [ 364.056421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.070665] LNet: 11512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.085725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.141750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 364.473071] hrtimer: interrupt took 2252491 ns [ 370.571492] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:01:35 (1786183295) [ 370.722640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.789752] LNet: 12097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.831788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.928509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 371.310491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 378.968304] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:01:43 (1786183303) [ 379.265589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.284511] LNet: 12688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.299947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.354806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 379.627709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.117953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 386.349510] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:01:50 (1786183310) [ 386.858966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.913686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.069080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 393.971787] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:01:58 (1786183318) [ 394.038245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.080143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.159076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 394.198892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.588238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 394.638436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.889878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.916952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.209296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.239978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.599638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.647766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.044150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.104373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.564226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.594805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.963953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 396.985680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.650869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.748850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.189839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 398.239970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.467137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 398.532409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.899829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.012901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.548413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 399.643247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.398267] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:02:12 (1786183332) [ 407.486441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.506791] LNet: 14543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.559490] LNet: 14543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 407.628724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.799042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 407.823721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 414.170981] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:02:19 (1786183339) [ 414.904415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.935929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.086555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 415.138209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.341313] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:02:26 (1786183346) [ 421.554351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.690462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.753380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 421.777539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 428.649744] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:02:33 (1786183353) [ 429.039433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.097431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.233425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 429.305539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 436.408714] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:02:41 (1786183361) [ 436.467650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.500419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.592050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 436.620110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 442.482238] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:02:47 (1786183367) [ 442.575582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.621771] LNet: 17474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.628992] LNet: 17474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 442.650966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.756450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 442.790985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 449.550179] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:02:54 (1786183374) [ 449.859533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.881884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.950574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.233282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.491541] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:03:02 (1786183382) [ 457.824582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.851867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.061233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.221847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.718814] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:03:09 (1786183389) [ 465.219292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.289052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.482719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.623404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.675810] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:03:16 (1786183396) [ 471.832431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.894353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.355398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.545554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.036067] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:03:23 (1786183403) [ 479.220838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.275295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.369773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.429767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.921947] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:03:31 (1786183411) [ 486.184363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.240692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.474736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.553459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.579638] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:03:38 (1786183418) [ 493.653496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.750298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.824094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 499.434351] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:03:44 (1786183424) [ 499.690353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.733962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.798454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 506.037672] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:03:51 (1786183431) [ 506.329654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.363541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.444343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 506.819388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 512.905824] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:03:58 (1786183438) [ 513.272491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.289523] LNet: 23511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.293484] LNet: 23511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 513.313298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.354915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 513.377776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 520.885752] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:04:05 (1786183445) [ 520.950149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.976200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.040476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 521.065638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 527.626543] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:04:12 (1786183452) [ 527.834524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.961649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.040406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 528.067728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 534.271782] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:04:19 (1786183459) [ 534.647861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.698710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.752401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 534.880814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 540.988882] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:04:26 (1786183466) [ 541.287885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.320777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.383377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 541.456713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 541.489309] LNetError: 25868:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 541.536918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 541.571654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 547.461970] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:04:32 (1786183472) [ 547.651674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.678696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.797783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3RaTh3CXdE [ 547.868863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3RaTh3CXdE [ 548.011175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3RaTh3CXdE [ 556.117651] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:04:40 (1786183480) [ 556.208051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.260617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.333769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 556.361605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 556.406541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 562.769362] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 06:04:47 (1786183487) [ 562.877357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.011719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.571446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 563.606682] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 565.235216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 566.552112] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 568.448156] Key type lgssc registered [ 569.531443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 572.548307] LNetError: 27628:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.31@tcp99 has no available nets [ 574.685479] Key type lgssc unregistered [ 575.236112] LNetError: 28741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.258142] LNet: Removed LNI 192.168.203.31@tcp [ 575.833794] Key type .llcrypt unregistered [ 575.847572] Key type ._llcrypt unregistered [ 582.610263] Key type ._llcrypt registered [ 582.614200] Key type .llcrypt registered [ 583.840389] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:05:08 (1786183508) [ 584.251676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 584.266886] alg: No test for adler32 (adler32-zlib) [ 585.152558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.209671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 588.043894] LNet: Message forwarding will be enabled in 2s [ 591.136166] LNet: Message forwarding enabled [ 591.518929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.544990] LNet: Message forwarding will stop in 2s to 6s [ 595.437293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.562097] LNet: Message forwarding disabled [ 597.879081] LNet: Message forwarding will be enabled in 2s [ 601.699331] LNet: Message forwarding enabled [ 604.167720] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:05:29 (1786183529) [ 604.275753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.353264] LNet: 30133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.408767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.468885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 611.190909] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:05:36 (1786183536) [ 611.248540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.274438] LNet: 30716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.299606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.338619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 611.365281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 611.394772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 611.441597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 611.475954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 611.521909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 611.549392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.598790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 611.633180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 611.668774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 611.702390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 611.734726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 611.767318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 611.803506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 611.834843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 611.890206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 611.993351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 612.077484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 612.116247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 612.135844] LNetError: 30765:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 617.784517] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:05:42 (1786183542) [ 617.925485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.971444] LNet: 31333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.020290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.090576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.310047] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:05:48 (1786183548) [ 624.868496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg331-server [ 626.434583] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:05:51 (1786183551) [ 626.603381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.681440] LNet: 32266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.724662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.768760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.785200] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 627.443769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[32-36]@tcp [ 627.459893] LNet: 32399:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 627.519853] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 627.532157] LNetError: 32270:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.33@tcp has gone from up to down [ 628.529719] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 628.541258] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 629.549346] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 629.570980] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 631.589598] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 631.606375] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 633.768436] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:05:58 (1786183558) [ 635.487351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg331-server [ 635.708704] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 635.730016] LNetError: 32270:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 637.188328] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:06:02 (1786183562) [ 637.278897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.326708] LNet: 33330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.336397] LNetError: 33330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.381545] LNet: Removed LNI 192.168.203.31@tcp [ 637.429878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.470571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 637.505810] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 638.094465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[32-37/2]@tcp [ 638.114892] LNet: 33452:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 638.122267] LNet: 33452:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 638.145667] LNetError: 33339:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.32@tcp has gone from up to down [ 638.161259] LNetError: 33339:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 638.165067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[32-37/2]@tcp [ 644.362110] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:06:09 (1786183569) [ 644.532990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.575941] LNetError: 34039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.625444] LNet: Removed LNI 192.168.203.31@tcp [ 644.704288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.904355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 645.079291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.103576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.158676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.198850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.221560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.268684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 645.354369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.372874] LNet: 34083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.377324] LNet: 34083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 645.392733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.489161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.546854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.580289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.631172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 645.695756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.721355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.772306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.833826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.859640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.894680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 645.951981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.982709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.038795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.126975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.176382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.243967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.335819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.371474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.435956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.634187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.672594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.709671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 646.752825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.780556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.828659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.171525] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:06:18 (1786183578) [ 655.049332] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg331-server [ 656.447296] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:06:21 (1786183581) [ 658.239201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg331-server [ 659.419820] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:06:24 (1786183584) [ 660.389652] Key type .llcrypt unregistered [ 660.391462] Key type ._llcrypt unregistered [ 665.618984] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 665.861794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 665.871996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 665.883793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 668.636082] Key type ._llcrypt registered [ 668.640261] Key type .llcrypt registered [ 668.820383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.841458] alg: No test for adler32 (adler32-zlib) [ 669.709665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.743903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.761750] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 670.223575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 670.238424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 672.324767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 672.342540] LNet: Removed LNI 10.1.2.3@tcp [ 672.606105] LNet: 36165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.610415] LNetError: 36165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.623592] LNet: Removed LNI 192.168.203.31@tcp [ 673.124534] Key type .llcrypt unregistered [ 673.126198] Key type ._llcrypt unregistered [ 684.248740] Key type ._llcrypt registered [ 684.250787] Key type .llcrypt registered [ 685.688098] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:06:50 (1786183610) [ 686.574371] Key type .llcrypt unregistered [ 686.576493] Key type ._llcrypt unregistered [ 692.755644] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.868881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.872630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.875636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.657806] Key type ._llcrypt registered [ 695.659860] Key type .llcrypt registered [ 695.904415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.920963] alg: No test for adler32 (adler32-zlib) [ 696.798693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.851618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.874060] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 697.553464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.572925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.000209] LNet: 37662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.011324] LNetError: 37662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 700.015829] LNet: Removed LNI 10.1.2.3@tcp [ 700.026822] LNet: Removed LNI 192.168.203.31@tcp [ 700.791502] Key type .llcrypt unregistered [ 700.797839] Key type ._llcrypt unregistered [ 712.704685] Key type ._llcrypt registered [ 712.706501] Key type .llcrypt registered [ 713.850839] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:07:18 (1786183638) [ 714.292469] Key type .llcrypt unregistered [ 714.299571] Key type ._llcrypt unregistered [ 719.599145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 719.696502] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 719.700276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 719.704367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 722.263574] Key type ._llcrypt registered [ 722.265509] Key type .llcrypt registered [ 722.414724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 722.431398] alg: No test for adler32 (adler32-zlib) [ 723.298226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.337399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 723.354210] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 725.080858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 725.097461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 727.267738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 727.293505] LNet: Removed LNI 10.1.2.3@tcp [ 727.306689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 727.326261] LNet: Removed LNI 10.1.2.31@tcp [ 727.337193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.356082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 728.078746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 728.095042] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 729.231434] LNet: 39586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.236785] LNetError: 39586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 729.241896] LNet: Removed LNI 10.1.2.31@tcp [ 729.250759] LNet: Removed LNI 10.1.2.3@tcp [ 730.194592] Key type .llcrypt unregistered [ 730.202558] Key type ._llcrypt unregistered [ 745.067181] Key type ._llcrypt registered [ 745.069885] Key type .llcrypt registered [ 746.971677] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:07:51 (1786183671) [ 747.663565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 747.679292] alg: No test for adler32 (adler32-zlib) [ 748.568560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.617771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.642513] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 749.752148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.776974] LNet: 40544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.782965] LNetError: 40544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.801252] LNet: Removed LNI 192.168.203.31@tcp [ 749.820203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.880715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.31@tcp [ 749.926320] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 750.817124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.31@tcp [ 750.864198] LNet: Removed LNI 192.168.203.31@tcp [ 751.014653] LNet: 40679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.501438] Key type .llcrypt unregistered [ 752.506618] Key type ._llcrypt unregistered [ 763.652381] Key type ._llcrypt registered [ 763.654081] Key type .llcrypt registered [ 765.283494] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:08:09 (1786183689) [ 767.314741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg331-server [ 768.918746] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:08:14 (1786183694) [ 769.796640] Key type .llcrypt unregistered [ 769.802233] Key type ._llcrypt unregistered [ 776.143552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.510207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.512564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.520949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.145730] Key type ._llcrypt registered [ 779.151334] Key type .llcrypt registered [ 779.532626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.589440] alg: No test for adler32 (adler32-zlib) [ 780.540905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.610158] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 781.281661] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 783.284862] LNet: 42519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.289609] LNetError: 42519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.294420] LNet: Removed LNI 192.168.203.31@tcp1 [ 783.304143] LNet: Removed LNI 10.1.2.3@tcp2 [ 784.326458] Key type .llcrypt unregistered [ 784.328550] Key type ._llcrypt unregistered [ 796.474351] Key type ._llcrypt registered [ 796.476194] Key type .llcrypt registered [ 797.686888] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:08:42 (1786183722) [ 798.113300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.124342] alg: No test for adler32 (adler32-zlib) [ 799.124745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.174969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.195605] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 799.989835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.007191] LNet: 43493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.011995] LNetError: 43493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.026629] LNet: Removed LNI 192.168.203.31@tcp [ 800.038568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.083506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 800.099207] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 806.562586] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:08:51 (1786183731) [ 806.659101] LNet: 44083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.663937] LNetError: 44083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.684883] LNet: Removed LNI 192.168.203.31@tcp [ 807.186429] Key type .llcrypt unregistered [ 807.190823] Key type ._llcrypt unregistered [ 812.436772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.706980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.734265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.741094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.442222] Key type ._llcrypt registered [ 815.443916] Key type .llcrypt registered [ 815.560307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.572900] alg: No test for adler32 (adler32-zlib) [ 816.445626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.486312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.502606] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 817.233100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 817.247190] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.658632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.678112] LNet: 44831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.683248] LNetError: 44831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 819.690600] LNet: Removed LNI 10.1.2.3@tcp [ 819.699829] LNet: Removed LNI 192.168.203.31@tcp [ 819.714449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.974268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 820.006260] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 820.013791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.465376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.490448] LNet: 44858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.494663] LNetError: 44858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.499041] LNetError: 44858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.503841] LNet: Removed LNI 10.1.2.3@tcp [ 820.512839] LNet: Removed LNI 192.168.203.31@tcp [ 820.541720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.667829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 820.704750] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 820.714580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.449840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 821.463585] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 822.407516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.422360] LNet: 45020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.428754] LNetError: 45020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 822.435278] LNetError: 45020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 822.441774] LNet: Removed LNI 10.1.2.3@tcp [ 822.445830] LNet: Removed LNI 192.168.203.31@tcp [ 822.457528] LNet: Removed LNI 192.168.203.31@tcp2 [ 822.513540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.593805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 822.723357] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 822.728658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.736809] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 830.189542] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:09:15 (1786183755) [ 830.295066] LNet: 45609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.302067] LNetError: 45609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.311311] LNetError: 45609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 830.318733] LNet: Removed LNI 10.1.2.3@tcp [ 830.322763] LNet: Removed LNI 192.168.203.31@tcp [ 830.344484] LNet: Removed LNI 192.168.203.31@tcp2 [ 831.302883] Key type .llcrypt unregistered [ 831.304772] Key type ._llcrypt unregistered [ 838.197271] Key type ._llcrypt registered [ 838.200620] Key type .llcrypt registered [ 838.320151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.329949] alg: No test for adler32 (adler32-zlib) [ 839.462598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.554238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.574423] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 840.152610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.168720] LNet: 46016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.173233] LNetError: 46016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.184950] LNet: Removed LNI 192.168.203.31@tcp [ 840.198556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.251635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.270506] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 840.658172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.681440] LNet: 46046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.689992] LNetError: 46046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.704955] LNet: Removed LNI 192.168.203.31@tcp [ 840.723819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.860548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.885619] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 841.206810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.236389] LNet: Removed LNI 192.168.203.31@tcp [ 841.268726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.314648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.332512] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 841.670826] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.779746] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.790502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.808356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 844.823950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.840716] LNet: 46158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.846516] LNet: 46158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 844.852887] LNetError: 46158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.859650] LNetError: 46158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 844.891988] LNet: Removed LNI 192.168.203.31@tcp [ 844.903667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.003420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 845.094948] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 851.928984] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:09:36 (1786183776) [ 852.099574] LNet: 46756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.104701] LNetError: 46756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.121531] LNet: Removed LNI 192.168.203.31@tcp [ 852.703284] Key type .llcrypt unregistered [ 852.705094] Key type ._llcrypt unregistered [ 859.289512] Key type ._llcrypt registered [ 859.291385] Key type .llcrypt registered [ 859.400907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.410878] alg: No test for adler32 (adler32-zlib) [ 860.278140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.320750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.351799] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 861.625968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.641416] LNet: 47158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.646656] LNetError: 47158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.656735] LNet: Removed LNI 192.168.203.31@tcp [ 861.674840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.732499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 861.752834] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 868.989827] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:09:53 (1786183793) [ 869.193763] LNet: 47743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.207366] LNetError: 47743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.240360] LNet: Removed LNI 192.168.203.31@tcp [ 870.195496] Key type .llcrypt unregistered [ 870.202304] Key type ._llcrypt unregistered [ 875.898705] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.216280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.220425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.227765] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.756531] Key type ._llcrypt registered [ 878.762207] Key type .llcrypt registered [ 879.193878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.205912] alg: No test for adler32 (adler32-zlib) [ 880.151531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.253516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.286615] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 880.929336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.945368] LNet: 48205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.949232] LNetError: 48205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.962772] LNet: Removed LNI 192.168.203.31@tcp [ 880.973981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.016456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 881.035797] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 888.810723] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:10:13 (1786183813) [ 888.919923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.973161] LNet: 48800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.990504] LNetError: 48800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.015139] LNet: Removed LNI 192.168.203.31@tcp [ 889.090264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.229454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 894.549782] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:10:19 (1786183819) [ 894.846908] LNet: 49386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.873696] Key type .llcrypt unregistered [ 895.878813] Key type ._llcrypt unregistered [ 902.092850] Key type ._llcrypt registered [ 902.095705] Key type .llcrypt registered [ 902.346434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.359929] alg: No test for adler32 (adler32-zlib) [ 903.267413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.297362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.312599] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 903.940378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.975677] LNet: 49788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.984299] LNetError: 49788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.000928] LNet: Removed LNI 192.168.203.31@tcp [ 904.035561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.088811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 904.113324] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 910.876162] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:10:35 (1786183835) [ 910.998928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.079120] LNet: 50375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.085136] LNetError: 50375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.156824] LNet: Removed LNI 192.168.203.31@tcp [ 911.235754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.504575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 917.790247] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:10:42 (1786183842) [ 919.112836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 920.708644] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:10:46 (1786183846) [ 922.533442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 924.455802] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:10:49 (1786183849) [ 926.219786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 927.568440] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:10:53 (1786183853) [ 928.981347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 930.632382] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:10:55 (1786183855) [ 932.728683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 934.292123] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:10:59 (1786183859) [ 934.368646] LNet: 52426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.890639] Key type .llcrypt unregistered [ 934.900276] Key type ._llcrypt unregistered [ 941.667039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 941.757393] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 941.762450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 941.779216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.654674] Key type ._llcrypt registered [ 943.658489] Key type .llcrypt registered [ 943.802949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.812095] alg: No test for adler32 (adler32-zlib) [ 944.768995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.824681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.847462] LNet: Added LNI 192.168.203.31@tcp [8/512/0/180] [ 944.853485] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 944.855676] LNet: 52747:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.984518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.001291] LNet: 52766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.006896] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.011729] LNet: Removed LNI 10.1.2.3@tcp [ 945.013965] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.033207] LNet: Removed LNI 192.168.203.31@tcp [ 945.051799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.121654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.149081] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 945.195728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.212208] LNet: 52790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.217733] LNetError: 52790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.235920] LNet: Removed LNI 192.168.203.31@tcp [ 945.263922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.371913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.401930] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 945.409668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.415927] LNet: 52798:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 945.564127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.595712] LNet: 52816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.602870] LNetError: 52816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.609836] LNet: Removed LNI 10.1.2.3@tcp [ 945.614518] LNetError: 52816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.632283] LNet: Removed LNI 192.168.203.31@tcp [ 945.654652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.734900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.762941] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 945.772858] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.783418] LNet: 52824:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 946.095714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.112477] LNet: 52850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.117675] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.123730] LNet: Removed LNI 10.1.2.3@tcp [ 946.125950] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.141712] LNet: Removed LNI 192.168.203.31@tcp [ 946.155809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.203103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 946.223793] LNet: Added LNI 192.168.203.31@tcp [8/256/0/182] [ 946.228509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 946.329909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.435146] LNet: 52878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.456797] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.463135] LNet: Removed LNI 10.1.2.3@tcp [ 946.466185] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.483257] LNet: Removed LNI 192.168.203.31@tcp [ 946.505330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.558448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 946.575582] LNet: Added LNI 192.168.203.31@tcp [8/256/0/182] [ 946.582185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 946.585689] LNet: 52886:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 946.640894] LNet: Removed LNI 10.1.2.3@tcp [ 946.656691] LNet: Removed LNI 192.168.203.31@tcp [ 947.305814] Key type .llcrypt unregistered [ 947.319264] Key type ._llcrypt unregistered [ 959.178503] Key type ._llcrypt registered [ 959.181575] Key type .llcrypt registered [ 960.159276] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 06:11:25 (1786183885) [ 962.052971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 964.328564] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:11:28 (1786183888) [ 964.811653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.835317] alg: No test for adler32 (adler32-zlib) [ 965.746619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.805738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.824787] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 966.796985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.31 [ 966.819961] LNet: 54155:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 966.828291] LNet: Removed LNI 192.168.203.31@tcp [ 973.326686] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:11:38 (1786183898) [ 973.476353] LNet: 54717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.486045] LNetError: 54717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.511204] LNet: Removed LNI 192.168.203.31@tcp [ 974.276614] Key type .llcrypt unregistered [ 974.281773] Key type ._llcrypt unregistered [ 980.400450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.804335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 982.609662] Key type ._llcrypt registered [ 982.611575] Key type .llcrypt registered [ 982.876577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.088759] alg: No test for adler32 (adler32-zlib) [ 984.392749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 991.826722] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:11:56 (1786183916) [ 991.947213] LNet: 55626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.951215] LNetError: 55626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.967207] LNet: Removed LNI 10.1.2.3@tcp [ 992.641565] Key type .llcrypt unregistered [ 992.647203] Key type ._llcrypt unregistered [ 999.411125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.723581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.726931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.730595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.323567] Key type ._llcrypt registered [ 1002.326357] Key type .llcrypt registered [ 1002.631939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.645818] alg: No test for adler32 (adler32-zlib) [ 1003.513408] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.339847] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:12:14 (1786183934) [ 1011.170915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1012.474482] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:12:17 (1786183937) [ 1012.671428] LNet: 56820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.681591] LNetError: 56820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.714285] LNet: Removed LNI 10.1.2.3@tcp [ 1013.412613] Key type .llcrypt unregistered [ 1013.415914] Key type ._llcrypt unregistered [ 1019.237024] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.560429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.567595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.577383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.314934] Key type ._llcrypt registered [ 1021.317354] Key type .llcrypt registered [ 1021.535702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.545431] alg: No test for adler32 (adler32-zlib) [ 1022.487369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.539665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1030.546262] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:12:35 (1786183955) [ 1030.905265] LNet: 58052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.918662] LNetError: 58052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1030.931424] LNet: Removed LNI 10.1.2.3@tcp [ 1031.393600] Key type .llcrypt unregistered [ 1031.398091] Key type ._llcrypt unregistered [ 1037.616666] Key type ._llcrypt registered [ 1037.619574] Key type .llcrypt registered [ 1037.768921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.790860] alg: No test for adler32 (adler32-zlib) [ 1038.922854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1038.983374] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1045.383461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1045.925936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_interrupt -r 1 [ 1045.954847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_interrupt -r 1 [ 1046.016420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1046.171045] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1046.454993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.487306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.526873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.558183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.597224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.634732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.688077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.759700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.830865] LNet: 58494:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.877689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.972852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.051568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.112490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.141414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.201340] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1047.207573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.208328] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1047.267098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.022591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_dropped -r 1 [ 1048.110299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_dropped -r 1 [ 1048.163683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1048.479855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.517155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.603445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.671904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.727886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.762260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.813255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.845337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.866740] LNet: 58603:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.887838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.925310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.953916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.988716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.034920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.068092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.113332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.781209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_aborted -r 1 [ 1049.826955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_aborted -r 1 [ 1049.860237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1049.886333] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1049.889927] LNet: Skipped 1 previous similar message [ 1050.196946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.244939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.281196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.375716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.422801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.488866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.649363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.698889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.739695] LNet: 58712:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.788308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.839521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.893196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.931458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.950791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.003913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.071995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.568909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_no_route -r 1 [ 1051.603209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_no_route -r 1 [ 1051.660922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1051.873292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.954514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.031942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.059819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.085443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.109851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.142309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.166738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.191679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.220470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.256963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.291996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.323805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.355496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.394464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1053.068342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_timeout -r 1 [ 1053.126461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_timeout -r 1 [ 1053.150758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1053.169346] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1053.174125] LNet: Skipped 1 previous similar message [ 1053.498599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.557092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.592242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.615783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.642743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.673818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.700805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.727401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.747660] LNet: 58931:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.752919] LNet: 58931:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1053.767259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.798674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.823403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.850660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.877384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.904147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.929928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.339518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_error -r 1 [ 1054.369946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_error -r 1 [ 1054.402482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1054.635914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.668501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.703241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.780355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.824799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.854453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.877749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.898431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.923899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.945410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.968159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.990960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.014743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.051406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.093381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.283551] LNet: 59085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.307907] LNetError: 59085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.338239] LNet: Removed LNI 192.168.203.31@tcp [ 1055.901708] Key type .llcrypt unregistered [ 1055.907576] Key type ._llcrypt unregistered [ 1074.547743] Key type ._llcrypt registered [ 1074.550175] Key type .llcrypt registered [ 1075.885899] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:13:20 (1786184000) [ 1076.764489] Key type .llcrypt unregistered [ 1076.766643] Key type ._llcrypt unregistered [ 1083.937513] Key type ._llcrypt registered [ 1083.938937] Key type .llcrypt registered [ 1084.570997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.585862] alg: No test for adler32 (adler32-zlib) [ 1085.464970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1085.487564] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1092.375326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1102.149966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.173460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.188801] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1103.640751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_interrupt -r 1 [ 1103.682920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_interrupt -r 1 [ 1103.733449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_interrupt -r 1 [ 1103.800468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_interrupt -r 1 [ 1103.842750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_interrupt -r 1 [ 1103.874904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_interrupt -r 1 [ 1103.914780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_interrupt -r 1 [ 1103.970447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_interrupt -r 1 [ 1104.011424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1104.025599] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1104.030595] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1104.285773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.309736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.334776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.354660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.373462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.397782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.428702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.455732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.477561] LNet: 60559:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.497106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.538453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.571946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.601721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.633606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.672460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.708864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.308952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_dropped -r 1 [ 1105.399089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_dropped -r 1 [ 1105.480448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_dropped -r 1 [ 1105.509982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_dropped -r 1 [ 1105.536848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_dropped -r 1 [ 1105.566796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_dropped -r 1 [ 1105.612957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_dropped -r 1 [ 1105.648646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_dropped -r 1 [ 1105.683869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1105.698139] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1105.703175] LNet: Skipped 2 previous similar messages [ 1105.929098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.960559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.984341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.011551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.036549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.062686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.085951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.120487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.139315] LNet: 60680:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.162301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.221835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.260349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.295855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.326059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.354648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.389623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.844417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_aborted -r 1 [ 1106.876070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_aborted -r 1 [ 1106.906165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_aborted -r 1 [ 1106.944727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_aborted -r 1 [ 1106.974556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_aborted -r 1 [ 1107.003902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_aborted -r 1 [ 1107.036838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_aborted -r 1 [ 1107.067597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_aborted -r 1 [ 1107.101469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1107.119457] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1107.123852] LNet: Skipped 2 previous similar messages [ 1107.362784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.392776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.423273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.447311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.472886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.505840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.550122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.588936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.607329] LNet: 60801:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.629969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.712833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.751842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.825110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.868346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.912131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.934395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.410396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_no_route -r 1 [ 1108.442412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_no_route -r 1 [ 1108.467832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_no_route -r 1 [ 1108.500366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_no_route -r 1 [ 1108.535890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_no_route -r 1 [ 1108.574746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_no_route -r 1 [ 1108.599452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_no_route -r 1 [ 1108.638834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_no_route -r 1 [ 1108.685596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1108.939794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.980391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.059860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.123125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.173166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.195703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.240955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.318696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.364819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.390724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.421121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.444871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.476710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.500808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.530500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.081088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_timeout -r 1 [ 1110.140212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_timeout -r 1 [ 1110.175492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_timeout -r 1 [ 1110.199160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_timeout -r 1 [ 1110.225268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_timeout -r 1 [ 1110.252847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_timeout -r 1 [ 1110.284198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_timeout -r 1 [ 1110.327106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_timeout -r 1 [ 1110.353810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1110.369340] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1110.373664] LNet: Skipped 5 previous similar messages [ 1110.582990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.622536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.651808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.688314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.725404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.770799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.816501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.852134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.879473] LNet: 61044:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.884496] LNet: 61044:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1110.908108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.982501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.006697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.026838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.059762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.098603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.156433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.766823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_error -r 1 [ 1111.798911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_error -r 1 [ 1111.829734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_error -r 1 [ 1111.869973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_error -r 1 [ 1111.942865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_error -r 1 [ 1111.979942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_error -r 1 [ 1112.017838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_error -r 1 [ 1112.070667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_error -r 1 [ 1112.110258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1112.381069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.486581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.528484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.585025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.612651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.650431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.677312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.709713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.745752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.781669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.814725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.859635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.901840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.946787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.997456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.055151] LNet: 61246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.076783] LNetError: 61246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.095716] LNet: Removed LNI 192.168.203.31@tcp [ 1119.109828] LNet: Removed LNI 192.168.203.31@tcp1 [ 1119.822780] Key type .llcrypt unregistered [ 1119.824760] Key type ._llcrypt unregistered [ 1137.343121] Key type ._llcrypt registered [ 1137.345421] Key type .llcrypt registered [ 1138.411615] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:14:23 (1786184063) [ 1139.109472] Key type .llcrypt unregistered [ 1139.114429] Key type ._llcrypt unregistered [ 1144.576733] Key type ._llcrypt registered [ 1144.580530] Key type .llcrypt registered [ 1144.788600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.798025] alg: No test for adler32 (adler32-zlib) [ 1145.690216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.715971] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1152.278278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1153.156964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_dropped -r 1 [ 1153.212303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_dropped -r 1 [ 1153.344019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1153.375562] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1153.664673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.690896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.711302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.737952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.765721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.797367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.819596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.841594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.854598] LNet: 62505:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.866845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.894946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.916800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.943787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.977374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.007943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.043538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1154.890638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_error -r 1 [ 1154.940245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_error -r 1 [ 1154.966695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1154.986215] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1155.284433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.307342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.341708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.369926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.440843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.489236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.522629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.555740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.575979] LNet: 62615:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.591790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.617993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.688680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.726970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.748771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.779193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.832956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.645690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_timeout -r 1 [ 1156.756591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_timeout -r 1 [ 1156.782517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1156.798722] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1157.314453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.338923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.365763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.388417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.418416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.447425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.469694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.488579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.501342] LNet: 62724:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.512489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.543134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.576163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.613923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.641582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.663661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.692648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.990764] LNet: 62769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.997057] LNetError: 62769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.014842] LNet: Removed LNI 192.168.203.31@tcp [ 1158.533494] Key type .llcrypt unregistered [ 1158.535647] Key type ._llcrypt unregistered [ 1174.736712] Key type ._llcrypt registered [ 1174.739953] Key type .llcrypt registered [ 1176.040855] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:15:00 (1786184100) [ 1176.968385] Key type .llcrypt unregistered [ 1176.977238] Key type ._llcrypt unregistered [ 1182.721868] Key type ._llcrypt registered [ 1182.725643] Key type .llcrypt registered [ 1182.895762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1182.905342] alg: No test for adler32 (adler32-zlib) [ 1183.803980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1183.825316] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1190.533857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1199.076677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.226717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1199.248532] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1200.532644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_dropped -r 1 [ 1200.575391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e remote_dropped -r 1 [ 1200.607162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_dropped -r 1 [ 1200.644878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e remote_dropped -r 1 [ 1200.678855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e remote_dropped -r 1 [ 1200.722967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e remote_dropped -r 1 [ 1200.773740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e remote_dropped -r 1 [ 1200.803929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e remote_dropped -r 1 [ 1200.841768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1200.856127] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1200.863654] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.131@tcp [ 1201.116914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.169270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.213723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.246739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.280414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.307464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.336351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.367628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.385729] LNet: 64241:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.400462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.428667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.479453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.517389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.552778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.580781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.638110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1201.668750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.337586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_error -r 1 [ 1202.367305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e remote_error -r 1 [ 1202.395849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_error -r 1 [ 1202.454734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e remote_error -r 1 [ 1202.476355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e remote_error -r 1 [ 1202.500849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e remote_error -r 1 [ 1202.529630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e remote_error -r 1 [ 1202.610682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e remote_error -r 1 [ 1202.654281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1202.697218] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1202.700714] LNet: Skipped 2 previous similar messages [ 1203.094961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.122681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.151469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.192086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.241603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.273925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.300341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.323511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.337222] LNet: 64364:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.352624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.372662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.396436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.416264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.440889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.471950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.494393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.525278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.040322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_timeout -r 1 [ 1204.066168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e remote_timeout -r 1 [ 1204.099868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_timeout -r 1 [ 1204.129185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e remote_timeout -r 1 [ 1204.155214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e remote_timeout -r 1 [ 1204.208535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e remote_timeout -r 1 [ 1204.259996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e remote_timeout -r 1 [ 1204.296117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e remote_timeout -r 1 [ 1204.335623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.131@tcp [ 1204.355472] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1204.562575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.600262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.639687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.673485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.727552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.755981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.801728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.835647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.857965] LNet: 64488:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.881917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.927723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.969468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.003281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.103598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.136793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.160284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.190929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.539793] LNet: 64569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.548519] LNetError: 64569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.554616] LNet: Removed LNI 192.168.203.31@tcp [ 1209.561705] LNet: Removed LNI 192.168.203.31@tcp1 [ 1209.849727] Key type .llcrypt unregistered [ 1209.854586] Key type ._llcrypt unregistered [ 1227.196031] Key type ._llcrypt registered [ 1227.199443] Key type .llcrypt registered [ 1228.772685] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:15:53 (1786184153) [ 1230.522621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1231.785416] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:15:57 (1786184157) [ 1232.475402] Key type .llcrypt unregistered [ 1232.477684] Key type ._llcrypt unregistered [ 1238.808730] Key type ._llcrypt registered [ 1238.812212] Key type .llcrypt registered [ 1239.001357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.015833] alg: No test for adler32 (adler32-zlib) [ 1239.942439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1239.964343] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1247.608250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1248.294980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e network_timeout -r 1 [ 1248.330822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e network_timeout -r 1 [ 1248.368147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1248.389816] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1248.671603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.789320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.865659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.889382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.942675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.982900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.017503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.042833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.055968] LNet: 66118:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.073723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.101211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.142264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.170389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.191124] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1249.206664] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1249.227844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.262857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.645160] LNet: 66161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.650584] LNetError: 66161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.679804] LNet: Removed LNI 192.168.203.31@tcp [ 1250.088543] Key type .llcrypt unregistered [ 1250.091824] Key type ._llcrypt unregistered [ 1269.086714] Key type ._llcrypt registered [ 1269.088792] Key type .llcrypt registered [ 1269.330629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.353825] alg: No test for adler32 (adler32-zlib) [ 1270.285642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.309176] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1277.192899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1285.100536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1285.129747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1285.156217] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1286.814546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e network_timeout -r 1 [ 1286.902919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e network_timeout -r 1 [ 1286.958936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e network_timeout -r 1 [ 1286.984510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e network_timeout -r 1 [ 1287.011761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e network_timeout -r 1 [ 1287.044144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e network_timeout -r 1 [ 1287.100718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e network_timeout -r 1 [ 1287.155501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e network_timeout -r 1 [ 1287.206486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1287.233416] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1287.454527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.479352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1287.503340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.527639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.549556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.651140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.671904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.697791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.718483] LNet: 67046:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.816898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.825950] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1287.840199] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.131@tcp [ 1287.881334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.916879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.950485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.979187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.010943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.069320] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1292.278158] LNet: 67124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.294413] LNetError: 67124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.336263] LNet: Removed LNI 192.168.203.31@tcp [ 1292.350771] LNet: Removed LNI 192.168.203.31@tcp1 [ 1292.632822] Key type .llcrypt unregistered [ 1292.639538] Key type ._llcrypt unregistered [ 1310.852115] Key type ._llcrypt registered [ 1310.855217] Key type .llcrypt registered [ 1312.060420] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:17:17 (1786184237) [ 1312.521469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.535343] alg: No test for adler32 (adler32-zlib) [ 1313.390936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.449616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1313.480310] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1314.184760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1314.202424] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1315.377722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 1315.461846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1315.538488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1315.577274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1315.613965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1315.652767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1315.686380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.31@tcp [ 1315.716329] LNetError: 68224:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.31@tcp to recovery [ 1316.577123] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1316.579082] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1318.624582] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1319.851203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1322.799720] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1323.764979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1323.794307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1323.861625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1323.879261] LNet: 68332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.884601] LNetError: 68332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.904107] LNet: Removed LNI 192.168.203.31@tcp [ 1324.906318] LNetError: 68332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.924290] LNet: Removed LNI 192.168.203.31@tcp1 [ 1324.945704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.996412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.015204] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1325.741337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.776237] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1326.657899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 1326.760391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.876128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1326.931840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1326.960938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1326.986107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1327.011669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1327.042752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.31@tcp [ 1327.064378] LNetError: 68604:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.31@tcp to recovery [ 1328.032338] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1331.025875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1334.249251] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1334.253872] LNet: Skipped 1 previous similar message [ 1339.352122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1339.378229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.408265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.627644] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:17:50 (1786184270) [ 1345.738461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.776226] LNet: 69321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.790349] LNetError: 69321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.796283] LNet: Removed LNI 192.168.203.31@tcp [ 1345.810148] LNet: Removed LNI 192.168.203.31@tcp1 [ 1345.827351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.860689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.877929] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1346.437162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.449920] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1347.110430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 1347.161889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1347.183901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1347.216305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1347.242373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1347.275296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1347.301248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.31@tcp --health 0 [ 1347.387265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1347.872151] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.31@tcp [ 1348.896364] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1348.899607] LNet: Skipped 1 previous similar message [ 1362.342962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1362.494317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1362.536677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.593564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.31@tcp --health 500 [ 1362.709553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1369.345273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1369.391324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.407251] LNet: 69806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.417379] LNetError: 69806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.421381] LNetError: 69806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1369.426283] LNet: Removed LNI 192.168.203.31@tcp [ 1369.436954] LNet: Removed LNI 192.168.203.31@tcp1 [ 1369.466551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.504798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.519995] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1370.092701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.108734] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1370.790824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 1370.822216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.879589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1370.937871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1370.987753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1371.056168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1371.111628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1371.134646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.31@tcp --health 0 [ 1372.576353] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1372.588041] LNet: Skipped 3 previous similar messages [ 1373.644986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1383.399424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1383.424435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1383.462511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1390.579370] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:18:35 (1786184315) [ 1391.563469] LNet: 70854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.571791] LNetError: 70854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.580737] LNetError: 70854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1391.589752] LNet: Removed LNI 192.168.203.31@tcp [ 1391.597983] LNet: Removed LNI 192.168.203.31@tcp1 [ 1392.457630] Key type .llcrypt unregistered [ 1392.461900] Key type ._llcrypt unregistered [ 1399.128987] Key type ._llcrypt registered [ 1399.132274] Key type .llcrypt registered [ 1399.301445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.321346] alg: No test for adler32 (adler32-zlib) [ 1400.196965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1400.216306] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1408.407104] Lustre: DEBUG MARKER: Initial discovery [ 1408.876657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.131@tcp [ 1410.914808] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1410.969707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1411.053983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.131@tcp [ 1411.122688] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1411.192662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1411.278336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.131@tcp -r 1 -m GET -l 3 [ 1411.369700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.131@tcp -r 1 -m GET -e local_error [ 1411.399714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.131@tcp -r 1 -m PUT -l 6 [ 1411.554497] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1412.619236] Lustre: DEBUG MARKER: Force oleg331-client.virtnet to discover 192.168.203.131@tcp (in background) [ 1412.960935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.131@tcp [ 1414.282797] Lustre: DEBUG MARKER: Force oleg331-server to discover 192.168.203.31@tcp [ 1416.069274] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1417.142203] Lustre: DEBUG MARKER: Wait for 71622 [ 1419.305737] Lustre: DEBUG MARKER: Finished wait on 71622 [ 1419.337053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1419.368417] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1419.375589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1419.691373] LNet: 72074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.703690] LNetError: 72074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.724843] LNet: Removed LNI 192.168.203.31@tcp [ 1420.003593] Key type .llcrypt unregistered [ 1420.005677] Key type ._llcrypt unregistered [ 1437.625456] Key type ._llcrypt registered [ 1437.631268] Key type .llcrypt registered [ 1438.724157] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:19:23 (1786184363) [ 1439.951429] Key type .llcrypt unregistered [ 1439.960330] Key type ._llcrypt unregistered [ 1447.016029] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1447.224067] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1447.236251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1447.254451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1450.388473] Key type ._llcrypt registered [ 1450.401272] Key type .llcrypt registered [ 1450.811502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.860704] alg: No test for adler32 (adler32-zlib) [ 1451.888214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.970378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.105082] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1453.013539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1453.060754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1461.478135] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:19:46 (1786184386) [ 1461.688384] LNet: 74158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.699549] LNetError: 74158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1461.706946] LNet: Removed LNI 10.1.2.3@tcp [ 1461.722298] LNet: Removed LNI 192.168.203.31@tcp [ 1462.372418] Key type .llcrypt unregistered [ 1462.374284] Key type ._llcrypt unregistered [ 1469.692447] Key type ._llcrypt registered [ 1469.695333] Key type .llcrypt registered [ 1470.001822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.012692] alg: No test for adler32 (adler32-zlib) [ 1470.946407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.989651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.008281] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1471.816344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1471.831045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1473.425419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.31@tcp [ 1474.464381] LNetError: Unexpected error -22 connecting to 192.168.203.31@tcp at host 192.168.203.31:988 [ 1479.868726] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:20:04 (1786184404) [ 1479.963509] LNet: 75286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.973482] LNetError: 75286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1479.983601] LNet: Removed LNI 10.1.2.3@tcp [ 1480.012631] LNet: Removed LNI 192.168.203.31@tcp [ 1480.821341] Key type .llcrypt unregistered [ 1480.825699] Key type ._llcrypt unregistered [ 1487.075484] Key type ._llcrypt registered [ 1487.077495] Key type .llcrypt registered [ 1487.193133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1487.209718] alg: No test for adler32 (adler32-zlib) [ 1488.084576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.117249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1488.185736] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1489.101876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1489.115992] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 1489.928637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.31@tcp1 --nid 192.168.203.31@tcp2 [ 1493.086260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp1 192.168.203.31@tcp2 [ 1493.103099] LNetError: 76133:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.31@tcp1 to 192.168.203.31@tcp2 - Route table may be misconfigured [ 1493.118500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp2 192.168.203.31@tcp1 [ 1499.430128] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:20:24 (1786184424) [ 1499.678995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.698967] LNet: 76706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.707901] LNetError: 76706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.718605] LNet: Removed LNI 192.168.203.31@tcp1 [ 1500.782879] LNet: Removed LNI 192.168.203.31@tcp2 [ 1500.800432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.841674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.857239] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1501.894140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1501.907953] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1502.610838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 1502.642581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.31@tcp [ 1502.664901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp1 192.168.203.31@tcp1 [ 1502.689941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.31@tcp -d 192.168.203.31@tcp -e network_timeout [ 1502.713563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -e network_timeout [ 1502.737449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -e network_timeout [ 1502.758180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -e network_timeout [ 1502.782412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1502.796764] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1502.811283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1502.881820] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1502.897259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1502.994406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1510.824275] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:20:35 (1786184435) [ 1511.001362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.030549] LNet: 77561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.040399] LNetError: 77561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.048577] LNetError: 77561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.057569] LNet: Removed LNI 192.168.203.31@tcp [ 1512.104590] LNet: Removed LNI 192.168.203.31@tcp1 [ 1512.122509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.169626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1512.479823] LNet: 77610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.157848] Key type .llcrypt unregistered [ 1513.159693] Key type ._llcrypt unregistered [ 1524.371848] Key type ._llcrypt registered [ 1524.373983] Key type .llcrypt registered [ 1525.326321] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:20:50 (1786184450) [ 1525.717234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.731297] alg: No test for adler32 (adler32-zlib) [ 1526.633831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.887890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1529.367725] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:20:54 (1786184454) [ 1529.601183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.616666] LNet: 78709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.636501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.687601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.709549] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1530.180825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1530.198239] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1531.212746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1531.250231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp1 [ 1531.287244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp1 [ 1536.847552] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:21:02 (1786184462) [ 1538.342971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg331-server [ 1539.638545] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:21:05 (1786184465) [ 1541.043826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg331-server [ 1543.205142] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:21:08 (1786184468) [ 1545.479729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg331-server [ 1547.257649] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:21:12 (1786184472) [ 1549.029989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg331-server [ 1550.722822] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:21:15 (1786184475) [ 1552.167136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg331-server [ 1553.488880] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:21:18 (1786184478) [ 1555.129550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg331-server [ 1556.483563] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:21:21 (1786184481) [ 1557.921494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg331-server [ 1559.331930] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:21:24 (1786184484) [ 1561.159845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg331-server [ 1562.850828] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:21:27 (1786184487) [ 1564.663621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg331-server [ 1567.081448] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:21:31 (1786184491) [ 1567.165215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.220391] LNet: 82580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.226725] LNetError: 82580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.247339] LNet: Removed LNI 192.168.203.31@tcp [ 1568.304600] LNet: Removed LNI 192.168.203.31@tcp1 [ 1568.382626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.505283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.583862] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1569.355660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1569.408874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1569.729661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.744310] LNet: 82734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.749153] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.753635] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.796238] LNet: Removed LNI 192.168.203.31@tcp [ 1570.812910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.854621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1570.874169] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1571.963110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1572.118340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.134524] LNet: 82880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.139805] LNetError: 82880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.158722] LNet: Removed LNI 192.168.203.31@tcp [ 1573.173956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.211645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.226620] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1573.716410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1573.781252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1575.209368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.225486] LNetError: 83036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.236296] LNet: Removed LNI 192.168.203.31@tcp [ 1575.251939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.299814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1575.323508] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1576.149549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1576.374716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.391522] LNet: 83181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.395733] LNet: 83181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1577.456470] LNet: Removed LNI 192.168.203.31@tcp [ 1577.481723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.534190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.553361] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1578.468993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1578.588445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1579.912752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.927947] LNetError: 83339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.932379] LNetError: 83339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1580.972264] LNet: Removed LNI 192.168.203.31@tcp [ 1580.996570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.054382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1581.095570] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1581.666436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1581.854101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.895665] LNet: Removed LNI 192.168.203.31@tcp [ 1582.946494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.016991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.033589] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1583.768569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1583.830976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1584.184795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.254488] LNet: Removed LNI 192.168.203.31@tcp [ 1585.301412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.410541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1585.445973] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1586.213229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1586.486598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.504389] LNet: 83777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.514679] LNet: 83777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1587.577460] LNet: Removed LNI 192.168.203.31@tcp [ 1587.604096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.684897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.732803] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1588.890691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1588.965321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1589.173322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.189263] LNetError: 83924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.195235] LNetError: 83924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1589.205752] LNet: Removed LNI 192.168.203.31@tcp [ 1589.222692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.270472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1589.337096] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1590.002932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1590.209115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.271491] LNet: Removed LNI 192.168.203.31@tcp [ 1591.286952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.320239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.333783] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1591.828361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1591.890776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1592.162640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.191938] LNet: Removed LNI 192.168.203.31@tcp [ 1593.206805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.249635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1593.265422] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1593.989855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1594.315405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.365883] LNet: Removed LNI 192.168.203.31@tcp [ 1595.405335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.443694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.477800] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1596.252418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1596.293077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.323316] LNet: Removed LNI 192.168.203.31@tcp [ 1596.341661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.379254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.399681] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1597.150152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1597.295837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.332908] LNet: Removed LNI 192.168.203.31@tcp [ 1597.352972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.385475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1597.469815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.514767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.578747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1597.694599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.723266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.761717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1597.837259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.869443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.910152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1597.931557] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1598.709772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.735255] LNet: Removed LNI 192.168.203.31@tcp [ 1598.753561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.828907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1606.027684] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:22:10 (1786184530) [ 1606.135489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.167095] LNet: 85378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.177926] LNet: 85378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1606.235639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.370474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1606.405918] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/180] [ 1607.549423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.571460] LNetError: 85522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.592874] LNetError: 85522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1607.652538] LNet: Removed LNI 192.168.203.31@tcp231 [ 1607.662973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.750946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1607.775564] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/180] [ 1607.779783] LNet: 85530:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.007983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1608.037834] LNet: Removed LNI 192.168.203.31@tcp231 [ 1608.049589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1608.153689] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/180] [ 1609.023735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.050135] LNet: Removed LNI 192.168.203.31@tcp231 [ 1609.065437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.132801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1609.168464] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/0] [ 1609.176448] LNet: 85682:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1609.556985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1609.601276] LNet: Removed LNI 192.168.203.31@tcp231 [ 1609.635432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1609.690238] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/0] [ 1610.633330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.662639] LNet: Removed LNI 192.168.203.31@tcp231 [ 1610.683979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.812732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1610.878883] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/60] [ 1610.892497] LNet: 85834:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.493702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1611.540728] LNet: Removed LNI 192.168.203.31@tcp231 [ 1611.562561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1611.610682] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/60] [ 1612.461665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.509043] LNet: Removed LNI 192.168.203.31@tcp231 [ 1612.528830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.588733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1612.606040] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/180] [ 1613.016809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1613.037270] LNet: Removed LNI 192.168.203.31@tcp231 [ 1613.054920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1613.084118] LNet: Added LNI 192.168.203.31@tcp231 [8/256/0/180] [ 1620.567563] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:22:24 (1786184544) [ 1620.788327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.861736] LNet: Removed LNI 192.168.203.31@tcp231 [ 1620.915906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.193811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1621.217078] LNet: Added LNI 192.168.203.31@tcp232 [8/256/0/180] [ 1622.297621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1622.345464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1622.375548] LNet: Removed LNI 192.168.203.31@tcp232 [ 1622.390387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1622.415334] LNet: Added LNI 192.168.203.31@tcp232 [8/256/0/180] [ 1623.007709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1623.044766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1623.083933] LNet: Removed LNI 192.168.203.31@tcp232 [ 1623.103441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1623.122155] LNet: Added LNI 192.168.203.31@tcp232 [8/256/0/180] [ 1623.807763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1623.838990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1623.864273] LNet: Removed LNI 192.168.203.31@tcp232 [ 1623.884495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1624.002071] LNet: Added LNI 192.168.203.31@tcp232 [8/256/0/180] [ 1625.051492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1625.094521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1625.153775] LNet: Removed LNI 192.168.203.31@tcp232 [ 1625.184381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1631.750802] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:22:36 (1786184556) [ 1632.948761] Key type .llcrypt unregistered [ 1632.950539] Key type ._llcrypt unregistered [ 1638.965532] Key type ._llcrypt registered [ 1638.977472] Key type .llcrypt registered [ 1639.277196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.285445] alg: No test for adler32 (adler32-zlib) [ 1640.222801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1640.247696] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1646.871408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1655.086348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.148402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1655.263481] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1656.765405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_interrupt -r 2 [ 1656.909954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_interrupt -r 2 [ 1656.986145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_interrupt -r 2 [ 1657.037246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_interrupt -r 2 [ 1657.083576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_interrupt -r 2 [ 1657.142410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_interrupt -r 2 [ 1657.309166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_interrupt -r 2 [ 1657.335087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_interrupt -r 2 [ 1657.361204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1657.400266] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1657.412259] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1657.795606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.843780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1657.895748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1657.921466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1657.948622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1657.978454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1658.004724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.037614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1658.053523] LNet: 88490:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1658.069624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1658.098738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1658.126599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.160264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1658.192208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1658.223977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.985712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_dropped -r 2 [ 1659.052643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_dropped -r 2 [ 1659.090130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_dropped -r 2 [ 1659.129374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_dropped -r 2 [ 1659.156566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_dropped -r 2 [ 1659.188269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_dropped -r 2 [ 1659.250071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_dropped -r 2 [ 1659.271285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_dropped -r 2 [ 1659.297625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1659.314569] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1659.580585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1659.603092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1659.626638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1659.648223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1659.669269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1659.694876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1659.718813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1659.742619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1659.755595] LNet: 88610:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.783214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1659.814372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1659.849467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1659.892262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1659.947169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.003240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1660.407493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_aborted -r 2 [ 1660.431830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_aborted -r 2 [ 1660.461628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_aborted -r 2 [ 1660.495278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_aborted -r 2 [ 1660.526111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_aborted -r 2 [ 1660.588463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_aborted -r 2 [ 1660.621274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_aborted -r 2 [ 1660.663352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_aborted -r 2 [ 1660.704740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1660.734919] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1660.978499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.035974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1661.071174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.097635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1661.114398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1661.137342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1661.155575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.172143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1661.184317] LNet: 88731:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.207453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1661.238333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1661.259606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1661.287754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1661.327504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1661.358605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.968850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_no_route -r 2 [ 1661.994893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_no_route -r 2 [ 1662.027123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_no_route -r 2 [ 1662.052500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_no_route -r 2 [ 1662.080308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_no_route -r 2 [ 1662.127704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_no_route -r 2 [ 1662.155737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_no_route -r 2 [ 1662.197449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_no_route -r 2 [ 1662.239883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1662.556125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1662.590235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1662.617755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1662.646686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1662.674906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1662.701611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1662.730462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1662.758160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1662.788796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1662.819852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1662.856659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1662.886852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.917813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1662.942879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.573787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e local_timeout -r 2 [ 1663.615058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e local_timeout -r 2 [ 1663.644871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e local_timeout -r 2 [ 1663.733573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e local_timeout -r 2 [ 1663.767497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e local_timeout -r 2 [ 1663.793343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e local_timeout -r 2 [ 1663.817950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e local_timeout -r 2 [ 1663.847470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e local_timeout -r 2 [ 1663.877288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1663.899184] LNet: There was an unexpected network error while writing to 192.168.203.131: rc = -22 [ 1663.909451] LNet: Skipped 1 previous similar message [ 1664.171946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1664.201837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1664.233331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1664.258966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1664.282517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1664.313694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1664.344262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1664.368623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1664.380871] LNet: 88971:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1664.384742] LNet: 88971:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1664.397133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1664.419561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1664.444672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1664.469344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1664.496293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1664.516823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1665.017774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp -m GET -e remote_dropped -r 2 [ 1665.050185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.131@tcp1 -m GET -e remote_dropped -r 2 [ 1665.078694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp -m GET -e remote_dropped -r 2 [ 1665.103442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp -d 192.168.203.31@tcp1 -m GET -e remote_dropped -r 2 [ 1665.132343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp -m GET -e remote_dropped -r 2 [ 1665.162439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.131@tcp1 -m GET -e remote_dropped -r 2 [ 1665.202832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp -m GET -e remote_dropped -r 2 [ 1665.251963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.31@tcp1 -d 192.168.203.31@tcp1 -m GET -e remote_dropped -r 2 [ 1665.277622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.31@tcp 192.168.203.131@tcp [ 1665.294381] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.131@tcp [ 1665.476348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1665.505783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1665.537106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1665.570841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1665.610702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1665.636662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1665.668355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1665.693978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1665.716517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1665.738679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1665.759238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1665.780251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1665.800874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1665.821793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1670.270354] LNet: 89169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.275973] LNetError: 89169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.284707] LNet: Removed LNI 192.168.203.31@tcp [ 1670.294237] LNet: Removed LNI 192.168.203.31@tcp1 [ 1670.626576] Key type .llcrypt unregistered [ 1670.630414] Key type ._llcrypt unregistered [ 1686.067689] Key type ._llcrypt registered [ 1686.073671] Key type .llcrypt registered [ 1687.341710] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:23:32 (1786184612) [ 1687.641860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1687.655554] alg: No test for adler32 (adler32-zlib) [ 1688.610676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.657900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.674482] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1689.419801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1689.440435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.31@tcp [ 1689.454401] LNetError: 90149:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.31@tcp to recovery [ 1689.534870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.31@tcp [ 1689.623579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.229850] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:23:41 (1786184621) [ 1696.477307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.492231] LNet: 90738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.496564] LNetError: 90738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.507344] LNet: Removed LNI 192.168.203.31@tcp [ 1696.524120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.559828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.578844] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1697.167978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1697.188992] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1697.928385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.31@tcp [ 1697.944663] LNetError: 91005:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.31@tcp to recovery [ 1698.601974] LNet: 1 local NIs in recovery (showing 1): 192.168.203.31@tcp [ 1698.603667] LNet: There was an unexpected network error while writing to 192.168.203.31: rc = -22 [ 1699.183173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.31@tcp [ 1707.140904] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:23:52 (1786184632) [ 1709.162223] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg331-server [ 1710.912456] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:23:55 (1786184635) [ 1712.488884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg331-server [ 1714.138072] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:23:59 (1786184639) [ 1716.106161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg331-server [ 1717.956474] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:24:02 (1786184642) [ 1718.106136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.159846] LNet: 92631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.203811] LNetError: 92631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.218169] LNet: Removed LNI 192.168.203.31@tcp [ 1718.249771] LNet: Removed LNI 192.168.203.31@tcp1 [ 1718.269751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.359419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.378321] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1719.095930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1719.189899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1719.249665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1719.328371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1719.376593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1719.417434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1725.203556] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:24:10 (1786184650) [ 1725.550351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.570333] LNet: 93360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.574664] LNetError: 93360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.588775] LNetError: 93360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1725.600937] LNet: Removed LNI 192.168.203.31@tcp [ 1725.611849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.656105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.675612] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1732.555509] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:24:17 (1786184657) [ 1732.675518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.706579] LNet: 94056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.721623] LNetError: 94056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.742259] LNet: Removed LNI 192.168.203.31@tcp [ 1732.773197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.869399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.888814] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1739.067380] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:24:24 (1786184664) [ 1740.276465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1741.562368] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:24:26 (1786184666) [ 1741.927643] LNet: 95098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.934560] LNetError: 95098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.948935] LNet: Removed LNI 192.168.203.31@tcp [ 1742.533595] Key type .llcrypt unregistered [ 1742.536519] Key type ._llcrypt unregistered [ 1747.738467] Key type ._llcrypt registered [ 1747.740738] Key type .llcrypt registered [ 1747.939569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.972731] alg: No test for adler32 (adler32-zlib) [ 1748.989692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1749.009572] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1755.876658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1760.597701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.131@tcp [ 1760.884927] LNet: 95540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.893751] LNetError: 95540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.909675] LNet: Removed LNI 192.168.203.31@tcp [ 1761.439509] Key type .llcrypt unregistered [ 1761.441828] Key type ._llcrypt unregistered [ 1773.700152] Key type ._llcrypt registered [ 1773.703738] Key type .llcrypt registered [ 1775.068884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1777.050298] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1779.172148] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:25:03 (1786184703) [ 1780.611952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1781.853243] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:25:07 (1786184707) [ 1783.713330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg331-server [ 1785.149596] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:25:10 (1786184710) [ 1786.797396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg331-server [ 1788.823224] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:25:13 (1786184713) [ 1790.396406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.494074] alg: No test for adler32 (adler32-zlib) [ 1791.557150] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 1791.886304] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1793.585974] Key type lgssc registered [ 1794.726180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1813.492426] Key type lgssc unregistered [ 1813.755122] LNet: 98720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.794204] LNetError: 98720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.832586] LNet: Removed LNI 192.168.203.31@tcp [ 1814.445683] Key type .llcrypt unregistered [ 1814.448554] Key type ._llcrypt unregistered [ 1829.354352] Key type ._llcrypt registered [ 1829.355569] Key type .llcrypt registered [ 1830.372524] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:25:55 (1786184755) [ 1831.044964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.053659] alg: No test for adler32 (adler32-zlib) [ 1832.059707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.114147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.157132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.197653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.248468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1839.567271] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:26:04 (1786184764) [ 1839.678274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.734556] LNet: 100128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.802142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.900715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.925677] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1840.761830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.776829] LNet: 100278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.782375] LNetError: 100278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.796076] LNet: Removed LNI 192.168.203.31@tcp [ 1840.812957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.849492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1840.866303] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1841.355797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.377978] LNetError: 100307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.392770] LNet: Removed LNI 192.168.203.31@tcp [ 1841.413964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.456856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1841.480598] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1847.256969] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:26:12 (1786184772) [ 1847.354323] LNet: 100892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.357357] LNet: 100892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1847.370669] LNetError: 100892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.401078] LNet: Removed LNI 192.168.203.31@tcp [ 1847.889332] Key type .llcrypt unregistered [ 1847.892215] Key type ._llcrypt unregistered [ 1853.779704] Key type ._llcrypt registered [ 1853.782689] Key type .llcrypt registered [ 1853.916173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1853.927994] alg: No test for adler32 (adler32-zlib) [ 1854.858242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.922597] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1854.925157] LNet: *** cfs_fail_loc=e003, val=100*** [ 1855.532965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1855.544717] LNet: *** cfs_fail_loc=e003, val=100*** [ 1855.547255] LNet: Skipped 2 previous similar messages [ 1855.552082] Ping reply received with mapping: [ 1855.553967] found metadata reply NI[192.168.203.31@tcp] [ 1855.557081] 1f cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.559875] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.562864] 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.651363] LNet: 101298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.658531] LNetError: 101298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.679467] LNet: Removed LNI 192.168.203.31@tcp [ 1857.457533] Key type .llcrypt unregistered [ 1857.460792] Key type ._llcrypt unregistered [ 1868.444796] Key type ._llcrypt registered [ 1868.450317] Key type .llcrypt registered [ 1869.804379] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:26:34 (1786184794) [ 1870.364661] Key type .llcrypt unregistered [ 1870.369075] Key type ._llcrypt unregistered [ 1877.700220] Key type ._llcrypt registered [ 1877.702226] Key type .llcrypt registered [ 1877.930344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1877.941330] alg: No test for adler32 (adler32-zlib) [ 1879.209476] LNetError: 102355:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1879.598481] Key type .llcrypt unregistered [ 1879.600302] Key type ._llcrypt unregistered [ 1884.647917] Key type ._llcrypt registered [ 1884.649778] Key type .llcrypt registered [ 1885.768428] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:26:50 (1786184810) [ 1886.185667] Key type .llcrypt unregistered [ 1886.189970] Key type ._llcrypt unregistered [ 1892.096894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1892.100730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1893.763192] Key type ._llcrypt registered [ 1893.765787] Key type .llcrypt registered [ 1894.003698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1894.016688] alg: No test for adler32 (adler32-zlib) [ 1894.887218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.953931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1905.078779] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:27:10 (1786184830) [ 1905.435313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.476477] LNet: 104136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.520954] LNetError: 104136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1905.542252] LNet: Removed LNI 10.1.2.3@tcp [ 1905.560654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.115614] Key type .llcrypt unregistered [ 1906.117465] Key type ._llcrypt unregistered [ 1911.709888] Key type ._llcrypt registered [ 1911.712188] Key type .llcrypt registered [ 1911.822432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1911.836610] alg: No test for adler32 (adler32-zlib) [ 1912.761396] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1912.871362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1913.013578] LNet: 104429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.017887] LNetError: 104429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.027777] LNet: Removed LNI 192.168.203.31@tcp [ 1913.531510] Key type .llcrypt unregistered [ 1913.533449] Key type ._llcrypt unregistered [ 1925.492234] Key type ._llcrypt registered [ 1925.494186] Key type .llcrypt registered [ 1927.335956] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:27:31 (1786184851) [ 1927.963437] Key type .llcrypt unregistered [ 1927.965691] Key type ._llcrypt unregistered [ 1934.865489] Key type ._llcrypt registered [ 1934.871114] Key type .llcrypt registered [ 1935.203681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1935.219846] alg: No test for adler32 (adler32-zlib) [ 1942.764706] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:27:47 (1786184867) [ 1943.103970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.138652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.210368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.235390] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1943.876919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.916750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1943.959990] LNet: Removed LNI 192.168.203.31@tcp [ 1944.204890] LNet: 106185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.572551] Key type .llcrypt unregistered [ 1944.574194] Key type ._llcrypt unregistered [ 1956.093572] Key type ._llcrypt registered [ 1956.095582] Key type .llcrypt registered [ 1957.423791] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:28:02 (1786184882) [ 1957.936551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.952951] alg: No test for adler32 (adler32-zlib) [ 1958.945439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.030796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.077802] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1960.044376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 1960.069839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.31@tcp [ 1960.084451] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1960.087806] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.090438] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 0 [ 1960.095134] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1960.097637] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1960.103807] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.105918] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 0 [ 1960.109750] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1960.111766] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.114245] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.116884] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 2010 [ 1960.126199] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1960.127794] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.131149] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.133927] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 2010 [ 1960.138063] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1965.950983] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:28:11 (1786184891) [ 1966.453693] LNet: 107725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.459559] LNetError: 107725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.473580] LNet: Removed LNI 192.168.203.31@tcp [ 1967.423780] Key type .llcrypt unregistered [ 1967.433166] Key type ._llcrypt unregistered [ 1972.508787] Key type ._llcrypt registered [ 1972.510823] Key type .llcrypt registered [ 1972.632925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1972.647396] alg: No test for adler32 (adler32-zlib) [ 1973.563937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1973.598924] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 1979.664935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 1988.354991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.405937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1988.457826] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 1989.460163] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1989.606601] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1989.619715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.632556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1992.051594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1992.085905] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2002.189558] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:28:47 (1786184927) [ 2002.496219] LNet: 109236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.507753] LNetError: 109236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.515083] LNet: Removed LNI 192.168.203.31@tcp [ 2002.518277] LNet: Removed LNI 192.168.203.31@tcp1 [ 2002.526433] LNet: Removed LNI 10.1.2.3@tcp99 [ 2003.517553] Key type .llcrypt unregistered [ 2003.525426] Key type ._llcrypt unregistered [ 2009.105615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2009.294825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2011.956569] Key type ._llcrypt registered [ 2011.962629] Key type .llcrypt registered [ 2012.153358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2012.166188] alg: No test for adler32 (adler32-zlib) [ 2013.061676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.100762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.117243] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2013.971769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2013.993629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2016.531538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2016.556911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.591107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2016.612334] LNetError: 109985:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2016.649602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.689363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2017.011929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2017.080881] LNet: 110005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.088881] LNetError: 110005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2017.098381] LNet: Removed LNI 10.1.2.3@tcp [ 2018.149865] LNet: Removed LNI 192.168.203.31@tcp [ 2018.749696] Key type .llcrypt unregistered [ 2018.761700] Key type ._llcrypt unregistered [ 2031.644919] Key type ._llcrypt registered [ 2031.656643] Key type .llcrypt registered [ 2033.408647] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:29:17 (1786184957) [ 2033.694655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.704100] alg: No test for adler32 (adler32-zlib) [ 2034.692972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.737435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.765322] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2035.577983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2035.635703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg331-client@tcp [ 2041.229298] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:29:26 (1786184966) [ 2041.380654] LNet: 111542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.385651] LNetError: 111542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.401248] LNet: Removed LNI 192.168.203.31@tcp [ 2042.121898] Key type .llcrypt unregistered [ 2042.124606] Key type ._llcrypt unregistered [ 2047.787431] Key type ._llcrypt registered [ 2047.800668] Key type .llcrypt registered [ 2048.000991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.014264] alg: No test for adler32 (adler32-zlib) [ 2048.998592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2049.015685] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2049.039984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2054.871256] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:29:40 (1786184980) [ 2056.503477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2058.516198] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:29:43 (1786184983) [ 2059.929477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2061.586801] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:29:46 (1786184986) [ 2061.841776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.859348] LNet: 112965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.865179] LNetError: 112965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.880712] LNet: Removed LNI 192.168.203.31@tcp [ 2061.909246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.975121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.999207] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2062.666845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2062.758608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2069.624226] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:29:54 (1786184994) [ 2070.095690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.181902] LNet: 113674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.203373] LNetError: 113674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.224480] LNet: Removed LNI 192.168.203.31@tcp [ 2070.238546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.274138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2070.330505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2071.006445] Key type .llcrypt unregistered [ 2071.012450] Key type ._llcrypt unregistered [ 2081.750172] Key type ._llcrypt registered [ 2081.792625] Key type .llcrypt registered [ 2082.977383] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:30:07 (1786185007) [ 2083.380261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.390866] alg: No test for adler32 (adler32-zlib) [ 2084.349587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.397361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.419382] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2085.526251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2085.567304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2093.021763] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:30:17 (1786185017) [ 2093.154346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.191877] LNet: 115233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.205979] LNetError: 115233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.236878] LNet: Removed LNI 192.168.203.31@tcp [ 2093.258798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.308721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2093.338343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2099.919431] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:30:24 (1786185024) [ 2099.998822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.024697] LNet: 115809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.048789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.131506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2100.345669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2100.375898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2100.445513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2100.817204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2100.843370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2100.899852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.921259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.980699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2101.294978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2101.326616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2101.374783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2101.789953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2101.823724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2108.724337] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:30:33 (1786185033) [ 2108.867953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.903557] LNet: 116448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.922513] LNet: 116448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2108.944764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.997156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.037280] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2109.631545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2109.662532] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2110.389993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2110.446620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2110.694905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2110.724281] LNetError: 116728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.729996] LNet: Removed LNI 192.168.203.31@tcp [ 2110.745536] LNet: Removed LNI 192.168.203.31@tcp2 [ 2110.765810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.829411] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2110.832844] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2110.836657] LNet: 116734:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2111.173996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2111.395447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.411720] LNet: 116771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.425094] LNet: 116771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2112.480359] LNet: Removed LNI 192.168.203.31@tcp [ 2112.486787] LNetError: 116771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.498363] LNetError: 116771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2112.522975] LNet: Removed LNI 192.168.203.31@tcp2 [ 2112.584476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.659207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.676178] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2113.527689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2113.568601] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2114.923740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2115.025386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2115.157926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2115.374351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.391610] LNetError: 117056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.448224] LNet: Removed LNI 192.168.203.31@tcp [ 2117.479723] LNet: Removed LNI 192.168.203.31@tcp2 [ 2117.498887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.541699] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2117.546630] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2117.551337] LNet: 117063:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.808510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2117.996761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.011665] LNet: 117100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.018382] LNet: 117100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2118.023570] LNet: Removed LNI 192.168.203.31@tcp [ 2119.078962] LNet: Removed LNI 192.168.203.31@tcp2 [ 2119.099780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.133604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.212915] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2120.482883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.496947] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2121.375796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2121.432201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2121.751695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.770765] LNetError: 117380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.776984] LNetError: 117380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2122.784134] LNet: Removed LNI 192.168.203.31@tcp [ 2122.793307] LNet: Removed LNI 192.168.203.31@tcp2 [ 2122.807196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.848424] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2122.853908] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2122.858651] LNet: 117386:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2123.407849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2123.651223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.705560] LNet: Removed LNI 192.168.203.31@tcp [ 2124.714650] LNet: Removed LNI 192.168.203.31@tcp2 [ 2124.728177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.794256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.816828] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2125.310383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2125.325852] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2126.276777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2126.302509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2126.341532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2126.562619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.577409] LNet: 117707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.586154] LNet: 117707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2127.648359] LNet: Removed LNI 192.168.203.31@tcp [ 2128.677744] LNet: Removed LNI 192.168.203.31@tcp2 [ 2128.693929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.735329] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2128.740865] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2128.744986] LNet: 117714:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2129.072905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2129.330292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.347634] LNet: Removed LNI 192.168.203.31@tcp [ 2130.444930] LNet: Removed LNI 192.168.203.31@tcp2 [ 2130.459759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.494664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.513195] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2131.214976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2131.337430] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2132.356989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2132.412794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2132.540707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2133.134825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.157853] LNetError: 118048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.165301] LNetError: 118048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2133.174155] LNet: Removed LNI 192.168.203.31@tcp [ 2133.203707] LNet: Removed LNI 192.168.203.31@tcp2 [ 2133.227274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.303583] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2133.310113] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2133.316137] LNet: 118054:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2133.801636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2134.408105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.458993] LNet: Removed LNI 192.168.203.31@tcp [ 2135.468911] LNet: Removed LNI 192.168.203.31@tcp2 [ 2135.482898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.523396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.549839] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2136.587440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2136.608045] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2137.392167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2137.453253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2137.481458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2137.560652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2137.994472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.104591] LNet: Removed LNI 192.168.203.31@tcp [ 2140.135241] LNet: Removed LNI 192.168.203.31@tcp2 [ 2140.152714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.211337] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2140.215567] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2140.748881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2147.595849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2149.268102] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:31:14 (1786185074) [ 2149.456599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.501380] LNet: 119168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.571782] LNet: 119168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2149.589581] LNetError: 119168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.607503] LNetError: 119168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2149.623134] LNet: Removed LNI 192.168.203.31@tcp [ 2150.709227] LNet: Removed LNI 192.168.203.31@tcp2 [ 2150.763914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.138966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.162506] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2152.114888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2152.132281] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.496202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.525580] LNet: Removed LNI 10.1.2.3@tcp [ 2153.539787] LNet: Removed LNI 192.168.203.31@tcp [ 2153.553547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.607256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2153.632932] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2153.637932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2154.038425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2154.053944] LNet: Removed LNI 192.168.203.31@tcp [ 2154.066734] LNet: Removed LNI 10.1.2.3@tcp [ 2161.172639] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 06:31:25 (1786185085) [ 2161.273820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.308581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.374885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.395307] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2162.355124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2162.390137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.434929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.462106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.498758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.541890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.566261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.593381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.620839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.646671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.673962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.31@tcp [ 2162.765815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2168.910829] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:31:33 (1786185093) [ 2169.169368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.220540] LNet: Removed LNI 192.168.203.31@tcp [ 2170.255922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.443121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2170.585997] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2170.590557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2170.594247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2172.919408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.002082] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2173.835100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2173.872274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2175.810837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2175.866410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2175.943562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.31@tcp -m PUT -e local_timeout -r 1 [ 2175.997777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2176.173635] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2176.186658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2176.482897] LNetError: 120851:0:(lib-move.c:4801:lnet_parse()) 192.168.203.31@tcp, src 192.168.203.31@tcp: Dropping PUT (error -108 looking up sender) [ 2177.515743] LNet: Removed LNI 10.1.2.3@tcp [ 2178.537338] LNet: Removed LNI 192.168.203.31@tcp [ 2178.942472] Key type .llcrypt unregistered [ 2178.944768] Key type ._llcrypt unregistered [ 2190.449919] Key type ._llcrypt registered [ 2190.454635] Key type .llcrypt registered [ 2191.859921] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:31:56 (1786185116) [ 2192.210293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.225559] alg: No test for adler32 (adler32-zlib) [ 2193.130958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.397977] LNet: 122166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.215548] Key type .llcrypt unregistered [ 2194.217666] Key type ._llcrypt unregistered [ 2201.130440] Key type ._llcrypt registered [ 2201.132385] Key type .llcrypt registered [ 2201.326987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2201.340871] alg: No test for adler32 (adler32-zlib) [ 2202.198625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2202.220278] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2209.955102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 2210.187810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2210.695888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.131@tcp [ 2210.873761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2210.898412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2210.919909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2210.943204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2210.969872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2210.999760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2211.025818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2211.056299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2211.074195] LNet: 122620:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.098283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2211.149091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2211.202418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2211.225990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2211.258915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2211.289595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2212.090422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2212.369399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.393930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.427367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.489216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.551352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.577133] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.131@tcp [ 2212.596679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.654322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.728453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.785588] LNet: 122737:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.853854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.876169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.902298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2212.925505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2212.954489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2212.983166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2214.058439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.131@tcp [ 2214.333255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2214.375778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2214.399048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2214.463792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2214.506589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2214.535424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2214.564501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2214.608562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2214.630328] LNet: 122854:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2214.652151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2214.688569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2214.721381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.752312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.793488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.834349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2215.742095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2216.049965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2216.111658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2216.166480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2216.214180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2216.256294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2216.295073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2216.319253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2216.362280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2216.403535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2216.520536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2216.679988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2216.767931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2216.794489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2216.839771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2217.475441] LNet: 123022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.487364] LNetError: 123022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.504798] LNet: Removed LNI 192.168.203.31@tcp [ 2218.250995] Key type .llcrypt unregistered [ 2218.257889] Key type ._llcrypt unregistered [ 2238.725169] Key type ._llcrypt registered [ 2238.729579] Key type .llcrypt registered [ 2240.224541] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:32:44 (1786185164) [ 2240.662475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.671942] alg: No test for adler32 (adler32-zlib) [ 2241.546870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.990307] LNet: 123910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.442388] Key type .llcrypt unregistered [ 2242.444043] Key type ._llcrypt unregistered [ 2250.027558] Key type ._llcrypt registered [ 2250.029890] Key type .llcrypt registered [ 2250.174345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2250.186563] alg: No test for adler32 (adler32-zlib) [ 2251.094437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2251.115962] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2258.903962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.131@tcp [ 2267.371836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.413795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2267.434971] LNet: Added LNI 192.168.203.31@tcp1 [8/256/0/180] [ 2268.183993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2268.809402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.131@tcp [ 2269.113933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.137955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2269.160748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2269.240417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2269.269735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2269.291860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2269.313812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2269.339981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2269.353909] LNet: 124566:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.371876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2269.396719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2269.424038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.450444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.472376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.498764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2270.481910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2270.714568] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.131@tcp, 192.168.203.131@tcp1 [ 2270.909606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.955799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.980649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2271.046767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2271.101896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2271.141715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2271.186913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2271.228776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2271.285992] LNet: 124684:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.322512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2271.356286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.408456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.454913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.486407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.524383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2272.085271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.131@tcp [ 2272.408757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.467303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2272.495926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.536790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2272.588874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2272.633591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2272.660923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2272.691547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2272.710633] LNet: 124801:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.726959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2272.753735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2272.778870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2272.801271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2272.834920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2272.881725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2273.923090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2274.218549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2274.251500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2274.288990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2274.322958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.350624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.390907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2274.478876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2274.552539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2274.573529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2274.595297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2274.616767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2274.638631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2274.663453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2274.684773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2280.051224] LNet: 125003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.056416] LNetError: 125003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.060779] LNet: Removed LNI 192.168.203.31@tcp [ 2280.068796] LNet: Removed LNI 192.168.203.31@tcp1 [ 2280.458811] Key type .llcrypt unregistered [ 2280.464456] Key type ._llcrypt unregistered [ 2297.804235] Key type ._llcrypt registered [ 2297.805987] Key type .llcrypt registered [ 2298.972824] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 06:33:43 (1786185223) [ 2299.605678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.614457] alg: No test for adler32 (adler32-zlib) [ 2300.504076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.549282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.613842] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2301.374580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2301.388650] LNet: Added LNI 192.168.203.31@tcp2 [8/256/0/180] [ 2302.322282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.31@tcp [ 2302.414751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2303.456275] LNet: Removed LNI 192.168.203.31@tcp [ 2309.106973] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 06:33:54 (1786185234) [ 2310.476853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2311.925828] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:33:57 (1786185237) [ 2317.640511] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:34:02 (1786185242) [ 2323.524047] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:34:08 (1786185248) [ 2330.034662] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:34:15 (1786185255) [ 2336.211201] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:34:21 (1786185261) [ 2342.481971] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:34:27 (1786185267) [ 2349.235375] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:34:34 (1786185274) [ 2355.262780] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:34:40 (1786185280) [ 2362.389938] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:34:47 (1786185287) [ 2368.004969] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:34:53 (1786185293) [ 2374.301323] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:34:59 (1786185299) [ 2381.508237] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:35:06 (1786185306) [ 2387.591555] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:35:12 (1786185312) [ 2394.642680] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:35:19 (1786185319) [ 2401.714110] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:35:26 (1786185326) [ 2409.000811] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:35:34 (1786185334) [ 2416.388457] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:35:41 (1786185341) [ 2423.195994] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:35:48 (1786185348) [ 2423.433617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.449563] LNet: 136695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.454665] LNetError: 136695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.468309] LNet: Removed LNI 192.168.203.31@tcp2 [ 2423.484868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.569966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2423.588761] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2424.466625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.481547] LNet: 136837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.486811] LNetError: 136837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.503165] LNet: Removed LNI 192.168.203.31@tcp [ 2424.521963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.566789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2424.586875] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2429.722916] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:35:54 (1786185354) [ 2436.308226] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:36:01 (1786185361) [ 2442.437449] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:36:07 (1786185367) [ 2449.357852] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:36:14 (1786185374) [ 2455.848295] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:36:21 (1786185381) [ 2462.887724] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:36:27 (1786185387) [ 2469.893579] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:36:35 (1786185395) [ 2477.461056] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:36:42 (1786185402) [ 2484.993382] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:36:50 (1786185410) [ 2491.776929] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:36:56 (1786185416) [ 2497.960613] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:37:03 (1786185423) [ 2503.687298] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:37:08 (1786185428) [ 2509.276767] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2253 sec ========== 06:37:14 (1786185434) [ 2509.771585] LNet: 144303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.779582] LNetError: 144303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.796880] LNet: Removed LNI 192.168.203.31@tcp [ 2510.385547] Key type .llcrypt unregistered [ 2510.387602] Key type ._llcrypt unregistered [ 2520.247745] Key type ._llcrypt registered [ 2520.249872] Key type .llcrypt registered [ 2520.636974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2520.652944] alg: No test for adler32 (adler32-zlib) [ 2521.764978] Lustre: Lustre: Build Version: 2.17.56_50_g80e3cdc [ 2521.932978] LNet: Added LNI 192.168.203.31@tcp [8/256/0/180] [ 2523.634771] Key type lgssc registered [ 2524.585880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.339150] Lustre: Mounted lustre-client - version 2.17.56_50_g80e3cdc [ 2577.439654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2583.177205] Lustre: Unmounted lustre-client [ 2619.953965] Key type lgssc unregistered [ 2620.134910] LNet: 147259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.138866] LNetError: 147259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.147434] LNet: Removed LNI 192.168.203.31@tcp [ 2620.780137] Key type .llcrypt unregistered [ 2620.781657] Key type ._llcrypt unregistered