[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 433117256 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002356] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006645] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010063] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.021660] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026813] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] *** VALIDATE tmpfs *** [ 0.031186] *** VALIDATE proc *** [ 0.032258] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038040] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.041948] debug: unmapping init [mem 0xffffffff8a059000-0xffffffff8a060fff] [ 0.043876] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049028] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052340] rcu: Hierarchical SRCU implementation. [ 0.054528] smp: Bringing up secondary CPUs ... [ 0.055635] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.059372] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.171154] node 0 deferred pages initialised in 106ms [ 0.175261] devtmpfs: initialized [ 0.177328] x86/mm: Memory block size: 128MB [ 0.181155] gcov: version magic: 0x41383552 [ 0.184430] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.185098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.187473] pinctrl core: initialized pinctrl subsystem [ 0.189311] [ 0.190000] ************************************************************* [ 0.193021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195018] ** ** [ 0.198024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201026] ** ** [ 0.203019] ** This means that this kernel is built to expose internal ** [ 0.206024] ** IOMMU data structures, which may compromise security on ** [ 0.208017] ** your system. ** [ 0.210020] ** ** [ 0.212022] ** If you see this message and you are not debugging the ** [ 0.215022] ** kernel, report this immediately to your vendor! ** [ 0.218020] ** ** [ 0.220021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223021] ************************************************************* [ 0.226793] NET: Registered protocol family 16 [ 0.228530] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.232090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.235097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.239122] cpuidle: using governor menu [ 0.240629] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.243631] PCI: Using configuration type 1 for base access [ 0.246138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.256113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.257024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.259073] cryptd: max_cpu_qlen set to 1000 [ 0.262261] ACPI: Added _OSI(Module Device) [ 0.264021] ACPI: Added _OSI(Processor Device) [ 0.267023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.270019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.274930] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.281487] ACPI: Interpreter enabled [ 0.283081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.284013] ACPI: Using IOAPIC for interrupt routing [ 0.286191] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.290488] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302558] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305074] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.311117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.317255] acpiphp: Slot [2] registered [ 0.318185] acpiphp: Slot [5] registered [ 0.320137] acpiphp: Slot [6] registered [ 0.322146] acpiphp: Slot [3] registered [ 0.323091] acpiphp: Slot [4] registered [ 0.325108] acpiphp: Slot [7] registered [ 0.326140] acpiphp: Slot [8] registered [ 0.327147] acpiphp: Slot [9] registered [ 0.329203] acpiphp: Slot [10] registered [ 0.331199] acpiphp: Slot [11] registered [ 0.333160] acpiphp: Slot [12] registered [ 0.335164] acpiphp: Slot [13] registered [ 0.336221] acpiphp: Slot [14] registered [ 0.338138] acpiphp: Slot [15] registered [ 0.339152] acpiphp: Slot [16] registered [ 0.341144] acpiphp: Slot [17] registered [ 0.343134] acpiphp: Slot [18] registered [ 0.345117] acpiphp: Slot [19] registered [ 0.346132] acpiphp: Slot [20] registered [ 0.348121] acpiphp: Slot [21] registered [ 0.350101] acpiphp: Slot [22] registered [ 0.352119] acpiphp: Slot [23] registered [ 0.353115] acpiphp: Slot [24] registered [ 0.355122] acpiphp: Slot [25] registered [ 0.357156] acpiphp: Slot [26] registered [ 0.359125] acpiphp: Slot [27] registered [ 0.360107] acpiphp: Slot [28] registered [ 0.362108] acpiphp: Slot [29] registered [ 0.364129] acpiphp: Slot [30] registered [ 0.365089] acpiphp: Slot [31] registered [ 0.367066] PCI host bridge to bus 0000:00 [ 0.368042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.371039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.378047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.381044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.383196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386199] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390591] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.399015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.403058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.406021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.409019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414324] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.415000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.417063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.421892] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426024] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.437016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.441016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446827] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.455015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473790] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480323] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517852] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.523371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.525489] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.528243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.534078] iommu: Default domain type: Passthrough [ 0.536460] SCSI subsystem initialized [ 0.538165] ACPI: bus type USB registered [ 0.540132] usbcore: registered new interface driver usbfs [ 0.542102] usbcore: registered new interface driver hub [ 0.545094] usbcore: registered new device driver usb [ 0.547189] pps_core: LinuxPPS API ver. 1 registered [ 0.549012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.553090] PTP clock support registered [ 0.555113] EDAC MC: Ver: 3.0.0 [ 0.557121] PCI: Using ACPI for IRQ routing [ 0.559769] NetLabel: Initializing [ 0.561015] NetLabel: domain hash size = 128 [ 0.563012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.565105] NetLabel: unlabeled traffic allowed by default [ 0.568163] vgaarb: loaded [ 0.569229] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.572014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.578003] clocksource: Switched to clocksource kvm-clock [ 0.693332] VFS: Disk quotas dquot_6.6.0 [ 0.695178] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.697773] *** VALIDATE ramfs *** [ 0.699058] *** VALIDATE hugetlbfs *** [ 0.700595] pnp: PnP ACPI init [ 0.703168] pnp: PnP ACPI: found 6 devices [ 0.724310] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.727822] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.730408] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.732826] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.735406] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.738024] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.740786] NET: Registered protocol family 2 [ 0.743440] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.748703] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.753566] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.759517] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.764099] TCP: Hash tables configured (established 65536 bind 65536) [ 0.767758] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.771667] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.774918] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.778400] NET: Registered protocol family 1 [ 0.781517] RPC: Registered named UNIX socket transport module. [ 0.783798] RPC: Registered udp transport module. [ 0.785617] RPC: Registered tcp transport module. [ 0.787624] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.789868] NET: Registered protocol family 44 [ 0.791763] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.794368] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.796978] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.799509] PCI: CLS 0 bytes, default 64 [ 0.801349] Unpacking initramfs... [ 2.194501] debug: unmapping init [mem 0xffff8b153cc64000-0xffff8b153ffcffff] [ 2.198754] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.201279] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.204487] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.728588] Initialise system trusted keyrings [ 2.731905] Key type blacklist registered [ 2.734408] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.744349] zbud: loaded [ 2.748738] *** VALIDATE nfs *** [ 2.750288] *** VALIDATE nfs4 *** [ 2.752023] pstore: using deflate compression [ 2.756429] Platform Keyring initialized [ 2.875227] NET: Registered protocol family 38 [ 2.877161] Key type asymmetric registered [ 2.878891] Asymmetric key parser 'x509' registered [ 2.880772] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.885208] io scheduler mq-deadline registered [ 2.887808] io scheduler kyber registered [ 2.889995] io scheduler bfq registered [ 2.893312] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.897558] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.901127] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.904824] ACPI: Power Button [PWRF] [ 2.910829] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.918513] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.928403] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.955448] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.983580] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.988114] Non-volatile memory driver v1.3 [ 2.990248] Linux agpgart interface v0.103 [ 3.030073] virtio_blk virtio1: [vda] 146608 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.033602] vda: detected capacity change from 0 to 75063296 [ 3.048573] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.053084] vdb: detected capacity change from 0 to 1073741824 [ 3.061607] libphy: Fixed MDIO Bus: probed [ 3.067858] usbcore: registered new interface driver usbserial_generic [ 3.070887] usbserial: USB Serial support registered for generic [ 3.073195] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.077055] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.078976] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.081418] mousedev: PS/2 mouse device common for all mice [ 3.084500] rtc_cmos 00:05: RTC can wake from S4 [ 3.087907] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.091967] rtc_cmos 00:05: registered as rtc0 [ 3.094431] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.095974] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.098140] intel_pstate: CPU model not supported [ 3.104145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.108743] hid: raw HID events driver (C) Jiri Kosina [ 3.110832] usbcore: registered new interface driver usbhid [ 3.113391] usbhid: USB HID core driver [ 3.115080] drop_monitor: Initializing network drop monitor service [ 3.117533] Initializing XFRM netlink socket [ 3.119588] NET: Registered protocol family 10 [ 3.122798] Segment Routing with IPv6 [ 3.124698] NET: Registered protocol family 17 [ 3.127153] mpls_gso: MPLS GSO support [ 3.133309] RAS: Correctable Errors collector initialized. [ 3.135610] AVX version of gcm_enc/dec engaged. [ 3.137498] AES CTR mode by8 optimization enabled [ 3.219526] sched_clock: Marking stable (3219500176, 0)->(4142289361, -922789185) [ 3.223776] registered taskstats version 1 [ 3.226215] Loading compiled-in X.509 certificates [ 3.228754] zswap: loaded using pool lzo/zbud [ 3.257822] Key type big_key registered [ 3.270511] Key type encrypted registered [ 3.272276] ima: No TPM chip found, activating TPM-bypass! [ 3.274265] ima: Allocated hash algorithm: sha1 [ 3.275838] ima: No architecture policies found [ 3.277601] evm: Initialising EVM extended attributes: [ 3.279643] evm: security.selinux [ 3.280608] evm: security.ima [ 3.281350] evm: security.capability [ 3.282420] evm: HMAC attrs: 0x1 [ 3.284789] rtc_cmos 00:05: setting system clock to 2026-08-26 23:13:17 UTC (1787785997) [ 3.290802] debug: unmapping init [mem 0xffffffff8b003000-0xffffffff8b1fffff] [ 3.294031] debug: unmapping init [mem 0xffffffff89d82000-0xffffffff8a058fff] [ 3.302183] Write protecting the kernel read-only data: 28672k [ 3.308244] debug: unmapping init [mem 0xffffffff88403000-0xffffffff885fffff] [ 3.311538] debug: unmapping init [mem 0xffffffff88d14000-0xffffffff88dfffff] [ 3.342742] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.352171] systemd[1]: Detected virtualization kvm. [ 3.354287] systemd[1]: Detected architecture x86-64. [ 3.356332] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.383201] systemd[1]: No hostname configured. [ 3.385069] systemd[1]: Set hostname to . [ 3.387311] random: systemd: uninitialized urandom read (16 bytes read) [ 3.389819] systemd[1]: Initializing machine ID from random generator. [ 3.419584] random: ln: uninitialized urandom read (6 bytes read) [ 3.515227] random: systemd: uninitialized urandom read (16 bytes read) [ 3.518050] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.523507] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.529198] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK [ 3.664296] random: fast init done ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.221681] device-mapper: uevent: version 1.0.3 [ 4.224238] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.009250] virtio_net virtio0 ens2: renamed from eth0 [ 5.131312] scsi host0: ata_piix [ 5.165870] scsi host1: ata_piix [ 5.167660] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.170422] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.839394] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.788153] random: crng init done [ 9.789870] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.284396] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.454726] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.697671] SELinux: Disabled at runtime. [ 11.761803] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.771454] systemd[1]: Detected virtualization kvm. [ 11.773526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.273249] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.278041] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.284464] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.288869] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.293518] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.303643] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.313125] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ 12.367074] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.739190] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.097765] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.125317] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.236221] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.249906] EDAC sbridge: Ver: 1.1.2 [ 14.157119] Key type dns_resolver registered [ 14.454413] NFS: Registering the id_resolver key type [ 14.456274] Key type id_resolver registered [ 14.458249] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-client login: [ 62.413022] libcfs: loading out-of-tree module taints kernel. [ 62.492333] Key type ._llcrypt registered [ 62.496064] Key type .llcrypt registered [ 63.057565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.072045] alg: No test for adler32 (adler32-zlib) [ 64.335870] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 64.928530] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 66.631202] Key type lgssc registered [ 67.841805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.353046] hrtimer: interrupt took 2460261 ns [ 191.322501] Lustre: Mounted lustre-client - version 2.17.57_53_g458ba6d [ 196.425143] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.895988] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing check_logdir /tmp/testlogs/ [ 215.328343] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing yml_node [ 217.055434] Lustre: lustre-OST0000-osc-ffff8b158321f000: disconnect after 24s idle [ 219.866301] Lustre: DEBUG MARKER: Client: 2.17.57.53 [ 222.426269] Lustre: DEBUG MARKER: MDS: 2.17.57.53 [ 225.719968] Lustre: DEBUG MARKER: OSS: 2.17.57.53 [ 227.778624] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 26 19:17:00 EDT 2026 [ 247.526704] Lustre: Unmounted lustre-client [ 270.086578] Key type lgssc unregistered [ 270.433421] LNet: 8406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.443465] LNetError: 8406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 270.457425] LNet: Removed LNI 192.168.204.40@tcp [ 271.217188] Key type .llcrypt unregistered [ 271.219697] Key type ._llcrypt unregistered [ 296.497986] Key type ._llcrypt registered [ 296.504118] Key type .llcrypt registered [ 296.636620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 296.650587] alg: No test for adler32 (adler32-zlib) [ 297.633620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 297.710932] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 297.742451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 301.713384] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 301.862627] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.870030] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.913800] LNet: Removed LNI 192.168.204.40@tcp [ 302.589385] Key type .llcrypt unregistered [ 302.591380] Key type ._llcrypt unregistered [ 308.107485] Key type ._llcrypt registered [ 308.110560] Key type .llcrypt registered [ 309.479142] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:18:22 (1787786302) [ 309.859811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.876157] alg: No test for adler32 (adler32-zlib) [ 310.812653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.909482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.940749] LNet: 9709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.973232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.037970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 311.066696] LNet: 9716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 317.455679] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:18:30 (1787786310) [ 317.582459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.609883] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.633353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.689483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 324.360911] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:18:37 (1787786317) [ 324.520789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.541979] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.586672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.699934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 331.317792] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:18:44 (1787786324) [ 331.443799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.476656] LNet: 11457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.507093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.608892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 338.165990] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:18:50 (1787786330) [ 338.298919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.327336] LNet: 12045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.355978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.462324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 338.962344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 345.244392] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:18:58 (1787786338) [ 345.418387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.456023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.532182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.033726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.436878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 352.715963] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:19:05 (1787786345) [ 352.938988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.974701] LNet: 13235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.983303] LNet: 13235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 353.016549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.092975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 359.392460] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:19:12 (1787786352) [ 359.551087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.598564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.691133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 359.739308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.110540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 360.147758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.556254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 360.596644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.088524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 361.126805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.716751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 361.765262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 362.231632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 362.272848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.709841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 362.750335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.211880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 363.254807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.653942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 363.715343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 364.046733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 364.082355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.499458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 364.527320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 365.045519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 365.089659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 365.418943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 365.452414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 370.802475] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:19:24 (1787786364) [ 371.041370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.061689] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.066568] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 371.091718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.158130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 371.182460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 376.047965] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:19:29 (1787786369) [ 376.126585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.162505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.220410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.246864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 381.339070] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:19:34 (1787786374) [ 381.434851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.490912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.577730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 381.614748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 388.407326] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:19:41 (1787786381) [ 388.740466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.848757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.012442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 389.079657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 396.304643] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:19:49 (1787786389) [ 396.481540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.603772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.693705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 396.780523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 403.240566] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:19:56 (1787786396) [ 403.442887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.465334] LNet: 17434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.471066] LNet: 17434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 403.487953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.564535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.619753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 409.879296] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:20:02 (1787786402) [ 409.960881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.006712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.075094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.600899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.913721] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:20:09 (1787786409) [ 417.111457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.160102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.367558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.609424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 423.132167] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:20:16 (1787786416) [ 423.192410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.225652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.368528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.668843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.790516] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:20:22 (1787786422) [ 429.997738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.047451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.453892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.735455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 437.630231] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:20:30 (1787786430) [ 437.740050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.817523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.960632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.008260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.961219] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:20:38 (1787786438) [ 446.404037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.466979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.585330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.657754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.678453] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:20:46 (1787786446) [ 453.891539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.955895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.044866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 460.681332] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:20:53 (1787786453) [ 460.774778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.859313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.965860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 466.939911] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:20:59 (1787786459) [ 467.078573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.152872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.268142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 467.650740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 473.515549] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:21:06 (1787786466) [ 473.691414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.714400] LNet: 23462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.719271] LNet: 23462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 473.734178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.794695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 473.826867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 479.938905] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:21:12 (1787786472) [ 480.049029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.095207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.247594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 480.291367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 486.953779] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:21:19 (1787786479) [ 487.045690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.106869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.225746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 487.286637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 493.499669] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:21:26 (1787786486) [ 493.651595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.746482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.926363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 493.995989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.946932] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:21:33 (1787786493) [ 501.039226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.106286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.214955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 501.296624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 501.314657] LNetError: 25820:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 501.358353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 501.397601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 508.822526] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:21:41 (1787786501) [ 509.240210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.311301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.401655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JRh4JE60Zn [ 509.492611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JRh4JE60Zn [ 509.603405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JRh4JE60Zn [ 516.839919] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:21:49 (1787786509) [ 517.132605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.168459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.232634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 517.281473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 517.371337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 524.169816] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:21:57 (1787786517) [ 524.498394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.526656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.579336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 524.601304] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 526.055333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 527.668519] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 529.663142] Key type lgssc registered [ 530.991976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.888211] LNetError: 27582:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.40@tcp99 has no available nets [ 538.016737] Key type lgssc unregistered [ 538.365462] LNetError: 28694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.386786] LNet: Removed LNI 192.168.204.40@tcp [ 539.104990] Key type .llcrypt unregistered [ 539.107725] Key type ._llcrypt unregistered [ 546.311704] Key type ._llcrypt registered [ 546.316681] Key type .llcrypt registered [ 547.606804] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:22:20 (1787786540) [ 548.197246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 548.231539] alg: No test for adler32 (adler32-zlib) [ 549.236607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.297376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 552.555081] LNet: Message forwarding will be enabled in 2s [ 555.615177] LNet: Message forwarding enabled [ 556.023798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.039626] LNet: Message forwarding will stop in 2s to 6s [ 560.554539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 561.638874] LNet: Message forwarding disabled [ 563.606302] LNet: Message forwarding will be enabled in 2s [ 566.751762] LNet: Message forwarding enabled [ 571.849560] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:22:44 (1787786564) [ 572.385966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.442661] LNet: 30102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.318979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.440245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 581.792220] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:22:54 (1787786574) [ 582.223921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.251814] LNet: 30684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.293752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.355980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 582.401523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 582.478679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 582.542385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 582.580180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 582.622287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 582.666959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.723671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 582.765781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 582.820164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 582.871946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 582.926499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 582.983728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 583.024434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 583.062682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 583.086174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 583.128660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 583.172770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 583.207821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 583.231851] LNetError: 30727:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 590.405872] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:23:02 (1787786582) [ 590.667608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.692724] LNet: 31299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.718220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.795210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.505479] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:23:11 (1787786591) [ 600.662939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg440-server [ 602.738755] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:23:15 (1787786595) [ 602.918594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.951332] LNet: 32230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.002128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.084800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.121809] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 604.062162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[41-45]@tcp [ 604.096983] LNet: 32363:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 604.161409] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 604.175378] LNetError: 32234:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.41@tcp has gone from up to down [ 605.167448] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 605.186414] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 606.192618] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 606.217958] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 608.230825] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 608.255895] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 611.168290] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:23:24 (1787786604) [ 612.329761] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 612.344084] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 612.989729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg440-server [ 615.090616] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:23:27 (1787786607) [ 615.208851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.241794] LNet: 33292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.249446] LNetError: 33292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.294951] LNet: Removed LNI 192.168.204.40@tcp [ 615.355778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.493875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.568910] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 616.634354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 616.647124] LNet: 33416:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 616.653022] LNet: 33416:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 616.675625] LNetError: 33303:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.41@tcp has gone from up to down [ 616.681575] LNetError: 33303:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 616.698224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 623.720858] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:23:36 (1787786616) [ 623.841596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.893662] LNet: 34002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.902489] LNetError: 34002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.961301] LNet: Removed LNI 192.168.204.40@tcp [ 624.035972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.200375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 624.311505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.396241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.462174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 624.516955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.568540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.642550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 624.732727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.789180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.907956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.001839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.055374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.148759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 625.269698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.327687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.443596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.577826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.624802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.716542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 625.820421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.859634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.944443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.041135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.078731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.132851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 626.231780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.290352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.350869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.447255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.493589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.579311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 626.661868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.744196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.861177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.289519] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:23:46 (1787786626) [ 635.596940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg440-server [ 637.438436] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:23:50 (1787786630) [ 640.111283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg440-server [ 642.050604] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:23:54 (1787786634) [ 642.339029] LNet: 35409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.344018] LNet: 35409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 643.232990] Key type .llcrypt unregistered [ 643.234802] Key type ._llcrypt unregistered [ 651.062607] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 651.177644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 651.181987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 651.192434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 653.949501] Key type ._llcrypt registered [ 653.954690] Key type .llcrypt registered [ 654.165861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.188826] alg: No test for adler32 (adler32-zlib) [ 655.158344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.244921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.304647] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 656.240982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 656.274496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 658.327376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 658.381758] LNet: Removed LNI 10.1.2.3@tcp [ 658.523603] LNet: 36135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.528181] LNetError: 36135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.539818] LNet: Removed LNI 192.168.204.40@tcp [ 659.190000] Key type .llcrypt unregistered [ 659.192744] Key type ._llcrypt unregistered [ 671.674413] Key type ._llcrypt registered [ 671.677859] Key type .llcrypt registered [ 673.448285] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:24:25 (1787786665) [ 674.081315] Key type .llcrypt unregistered [ 674.088556] Key type ._llcrypt unregistered [ 680.758373] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 680.972253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 680.994383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 681.010580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 683.803395] Key type ._llcrypt registered [ 683.805947] Key type .llcrypt registered [ 683.972343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.992540] alg: No test for adler32 (adler32-zlib) [ 685.000838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.079430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.115892] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 686.005783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 686.043402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.367558] LNet: 37636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.377285] LNetError: 37636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 689.384359] LNet: Removed LNI 10.1.2.3@tcp [ 689.394807] LNet: Removed LNI 192.168.204.40@tcp [ 690.127686] Key type .llcrypt unregistered [ 690.135044] Key type ._llcrypt unregistered [ 703.133928] Key type ._llcrypt registered [ 703.140625] Key type .llcrypt registered [ 704.829221] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:24:57 (1787786697) [ 705.628944] Key type .llcrypt unregistered [ 705.631707] Key type ._llcrypt unregistered [ 711.920871] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.079357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.090962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.096957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.702694] Key type ._llcrypt registered [ 714.705609] Key type .llcrypt registered [ 714.845777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.860547] alg: No test for adler32 (adler32-zlib) [ 715.788669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.858230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 715.886803] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 717.805567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.827528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.731619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 719.750644] LNet: Removed LNI 10.1.2.3@tcp [ 719.762038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 719.784280] LNet: Removed LNI 10.1.2.31@tcp [ 719.807728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 719.847973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.664753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 720.686199] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 721.495827] LNet: 39565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.502904] LNetError: 39565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 721.508274] LNet: Removed LNI 10.1.2.31@tcp [ 721.524580] LNet: Removed LNI 10.1.2.3@tcp [ 722.216699] Key type .llcrypt unregistered [ 722.218698] Key type ._llcrypt unregistered [ 733.451306] Key type ._llcrypt registered [ 733.456848] Key type .llcrypt registered [ 734.780618] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:25:27 (1787786727) [ 735.479162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.514810] alg: No test for adler32 (adler32-zlib) [ 736.470959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.523574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.551453] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 737.373245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.393595] LNet: 40523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.399894] LNetError: 40523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.419233] LNet: Removed LNI 192.168.204.40@tcp [ 737.436948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.487860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.40@tcp [ 737.512860] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 738.215161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.40@tcp [ 738.259440] LNet: Removed LNI 192.168.204.40@tcp [ 738.332391] LNet: 40659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.990368] Key type .llcrypt unregistered [ 738.993537] Key type ._llcrypt unregistered [ 748.852892] Key type ._llcrypt registered [ 748.856707] Key type .llcrypt registered [ 749.907463] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:25:42 (1787786742) [ 751.472352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg440-server [ 752.778235] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:25:45 (1787786745) [ 753.244261] Key type .llcrypt unregistered [ 753.249938] Key type ._llcrypt unregistered [ 758.592588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.767883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.771327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.775607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 761.298902] Key type ._llcrypt registered [ 761.301917] Key type .llcrypt registered [ 761.463302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.474657] alg: No test for adler32 (adler32-zlib) [ 762.351029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.415352] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 763.098486] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 764.952876] LNet: 42502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.960921] LNetError: 42502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.968558] LNet: Removed LNI 192.168.204.40@tcp1 [ 764.983196] LNet: Removed LNI 10.1.2.3@tcp2 [ 765.578072] Key type .llcrypt unregistered [ 765.580860] Key type ._llcrypt unregistered [ 775.778315] Key type ._llcrypt registered [ 775.780818] Key type .llcrypt registered [ 777.020524] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:26:09 (1787786769) [ 777.583618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.596078] alg: No test for adler32 (adler32-zlib) [ 778.498750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.553354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.594475] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 779.437745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.460644] LNet: 43477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.470854] LNetError: 43477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.483969] LNet: Removed LNI 192.168.204.40@tcp [ 779.506640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.563439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 779.587233] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 785.714556] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:26:18 (1787786778) [ 785.978778] LNet: 44068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.983819] LNetError: 44068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.004154] LNet: Removed LNI 192.168.204.40@tcp [ 786.578402] Key type .llcrypt unregistered [ 786.582515] Key type ._llcrypt unregistered [ 792.774614] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.922069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.926995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.934922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.657884] Key type ._llcrypt registered [ 794.659462] Key type .llcrypt registered [ 794.833772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.855187] alg: No test for adler32 (adler32-zlib) [ 795.730763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.787845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.812134] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 796.598472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.629377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.120830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.145269] LNet: 44800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.170149] LNetError: 44800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.176143] LNet: Removed LNI 10.1.2.3@tcp [ 799.189208] LNet: Removed LNI 192.168.204.40@tcp [ 799.245836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.381165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.402433] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 799.407812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.169538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.193571] LNet: 44827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.220557] LNetError: 44827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.229499] LNetError: 44827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.237563] LNet: Removed LNI 10.1.2.3@tcp [ 800.250166] LNet: Removed LNI 192.168.204.40@tcp [ 800.278302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.352705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.373522] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 800.390562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.929509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 800.952486] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 802.220304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.241581] LNet: 44989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.249714] LNetError: 44989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.255626] LNetError: 44989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.262136] LNet: Removed LNI 10.1.2.3@tcp [ 802.264482] LNet: Removed LNI 192.168.204.40@tcp [ 802.276861] LNet: Removed LNI 192.168.204.40@tcp2 [ 802.297983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.377488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.427032] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 802.437894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.452721] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 810.821393] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:26:43 (1787786803) [ 810.948974] LNet: 45580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.953973] LNetError: 45580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.959528] LNetError: 45580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 810.965256] LNet: Removed LNI 10.1.2.3@tcp [ 810.968474] LNet: Removed LNI 192.168.204.40@tcp [ 810.992101] LNet: Removed LNI 192.168.204.40@tcp2 [ 811.898257] Key type .llcrypt unregistered [ 811.915785] Key type ._llcrypt unregistered [ 818.039745] Key type ._llcrypt registered [ 818.042407] Key type .llcrypt registered [ 818.335739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.369834] alg: No test for adler32 (adler32-zlib) [ 819.319711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.397099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.447741] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 820.288323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.338288] LNet: 45985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.351641] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.385880] LNet: Removed LNI 192.168.204.40@tcp [ 820.432800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.548315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.577463] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 821.104313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.128691] LNet: 46015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.151261] LNetError: 46015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.166942] LNet: Removed LNI 192.168.204.40@tcp [ 821.182642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.261607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.283678] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 821.671545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.705841] LNet: Removed LNI 192.168.204.40@tcp [ 821.726745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.796671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.827582] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 822.376834] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.539151] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.542226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.555286] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.976904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.996719] LNet: 46122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.001692] LNet: 46122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 825.007685] LNetError: 46122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.012631] LNetError: 46122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 825.030272] LNet: Removed LNI 192.168.204.40@tcp [ 825.059586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.143985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 825.172806] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 831.325371] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:27:04 (1787786824) [ 831.428122] LNet: 46712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.437261] LNetError: 46712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.457362] LNet: Removed LNI 192.168.204.40@tcp [ 832.071428] Key type .llcrypt unregistered [ 832.076761] Key type ._llcrypt unregistered [ 837.880325] Key type ._llcrypt registered [ 837.882530] Key type .llcrypt registered [ 838.089357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.099786] alg: No test for adler32 (adler32-zlib) [ 839.022542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.108686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.140922] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 839.858297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.879209] LNet: 47121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.885835] LNetError: 47121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.899375] LNet: Removed LNI 192.168.204.40@tcp [ 839.929511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.007472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 840.052886] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 845.925385] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:27:19 (1787786839) [ 846.215374] LNet: 47714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.224767] LNetError: 47714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.238455] LNet: Removed LNI 192.168.204.40@tcp [ 846.836621] Key type .llcrypt unregistered [ 846.839322] Key type ._llcrypt unregistered [ 852.896874] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 853.044121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 853.051641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.079591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.764566] Key type ._llcrypt registered [ 855.766123] Key type .llcrypt registered [ 855.939268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.960483] alg: No test for adler32 (adler32-zlib) [ 856.865982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.925584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.945888] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 858.030769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.074430] LNet: 48161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.087152] LNetError: 48161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.123273] LNet: Removed LNI 192.168.204.40@tcp [ 858.164990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.268212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 858.290944] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 865.487773] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:27:38 (1787786858) [ 866.063619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.126082] LNet: 48760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.138562] LNetError: 48760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.178358] LNet: Removed LNI 192.168.204.40@tcp [ 866.237611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.391574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 873.323711] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:27:46 (1787786866) [ 873.563704] LNet: 49342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.478472] Key type .llcrypt unregistered [ 874.485476] Key type ._llcrypt unregistered [ 882.807892] Key type ._llcrypt registered [ 882.814717] Key type .llcrypt registered [ 883.146367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.175565] alg: No test for adler32 (adler32-zlib) [ 884.047544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.149306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.216502] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 885.055213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.073884] LNet: 49744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.086707] LNetError: 49744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.103771] LNet: Removed LNI 192.168.204.40@tcp [ 885.123407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.273705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 885.303410] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 894.106800] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:28:06 (1787786886) [ 894.415370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.466969] LNet: 50338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.476983] LNetError: 50338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.496234] LNet: Removed LNI 192.168.204.40@tcp [ 894.541215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.681686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 901.867824] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:28:14 (1787786894) [ 903.953666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 906.891444] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:28:18 (1787786898) [ 909.691784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 911.817408] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:28:24 (1787786904) [ 913.477903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 915.817918] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:28:28 (1787786908) [ 918.101458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 920.117439] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:28:32 (1787786912) [ 922.178657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 924.051794] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:28:36 (1787786916) [ 924.226080] LNet: 52385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.195802] Key type .llcrypt unregistered [ 925.199100] Key type ._llcrypt unregistered [ 931.279774] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.446916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.454739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.488978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.036232] Key type ._llcrypt registered [ 934.040522] Key type .llcrypt registered [ 934.239246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.251623] alg: No test for adler32 (adler32-zlib) [ 935.233650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.334362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 935.372585] LNet: Added LNI 192.168.204.40@tcp [8/512/0/180] [ 935.384886] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 935.392933] LNet: 52719:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 935.530175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.572171] LNet: 52738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.584132] LNetError: 52738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 935.590643] LNet: Removed LNI 10.1.2.3@tcp [ 935.593655] LNetError: 52738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.615674] LNet: Removed LNI 192.168.204.40@tcp [ 935.665196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.798248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 935.832795] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 935.924831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.946451] LNet: 52762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.954710] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.971330] LNet: Removed LNI 192.168.204.40@tcp [ 935.986259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.061514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.084957] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 936.089916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.093222] LNet: 52770:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 936.201794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.230197] LNet: 52788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.243148] LNetError: 52788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.251616] LNet: Removed LNI 10.1.2.3@tcp [ 936.254522] LNetError: 52788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.269190] LNet: Removed LNI 192.168.204.40@tcp [ 936.295886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.370516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.395334] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 936.399478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.402925] LNet: 52796:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 936.609301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.633514] LNet: 52822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.640697] LNetError: 52822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.648298] LNet: Removed LNI 10.1.2.3@tcp [ 936.651131] LNetError: 52822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.667420] LNet: Removed LNI 192.168.204.40@tcp [ 936.704991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.810733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.832774] LNet: Added LNI 192.168.204.40@tcp [8/256/0/182] [ 936.839723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 936.998854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.018384] LNet: 52850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.023706] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.028613] LNet: Removed LNI 10.1.2.3@tcp [ 937.030562] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.039715] LNet: Removed LNI 192.168.204.40@tcp [ 937.055491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.106982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.122885] LNet: Added LNI 192.168.204.40@tcp [8/256/0/182] [ 937.127181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 937.133213] LNet: 52858:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 937.188873] LNet: Removed LNI 10.1.2.3@tcp [ 937.195613] LNet: Removed LNI 192.168.204.40@tcp [ 937.914276] Key type .llcrypt unregistered [ 937.918122] Key type ._llcrypt unregistered [ 949.352925] Key type ._llcrypt registered [ 949.356764] Key type .llcrypt registered [ 950.594708] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:29:03 (1787786943) [ 952.055040] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 954.038368] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:29:06 (1787786946) [ 954.440264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.466861] alg: No test for adler32 (adler32-zlib) [ 955.375648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.422436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.453157] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 956.238885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.40 [ 956.253591] LNet: 54127:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 956.260287] LNet: Removed LNI 192.168.204.40@tcp [ 962.352141] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:29:15 (1787786955) [ 962.644050] LNet: 54697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.653507] LNetError: 54697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.664561] LNet: Removed LNI 192.168.204.40@tcp [ 963.200369] Key type .llcrypt unregistered [ 963.203982] Key type ._llcrypt unregistered [ 969.543180] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 969.790865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.475491] Key type ._llcrypt registered [ 972.478179] Key type .llcrypt registered [ 972.676160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.695911] alg: No test for adler32 (adler32-zlib) [ 973.934648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.342694] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:29:33 (1787786973) [ 980.801990] LNet: 55614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.816412] LNetError: 55614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.824119] LNet: Removed LNI 10.1.2.3@tcp [ 981.411404] Key type .llcrypt unregistered [ 981.413420] Key type ._llcrypt unregistered [ 987.455915] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.725405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.730245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.737059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 990.697666] Key type ._llcrypt registered [ 990.700557] Key type .llcrypt registered [ 990.984336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.023135] alg: No test for adler32 (adler32-zlib) [ 992.046615] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.631734] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:29:52 (1787786992) [ 1001.062338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1003.130138] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:29:55 (1787786995) [ 1003.294995] LNet: 56799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.314132] LNetError: 56799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.336121] LNet: Removed LNI 10.1.2.3@tcp [ 1004.118923] Key type .llcrypt unregistered [ 1004.122460] Key type ._llcrypt unregistered [ 1011.402916] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1011.673642] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1011.679151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1011.686699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1014.478641] Key type ._llcrypt registered [ 1014.486428] Key type .llcrypt registered [ 1014.775162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.803248] alg: No test for adler32 (adler32-zlib) [ 1015.892298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.027207] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1026.409630] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:30:18 (1787787018) [ 1026.991835] LNet: 58039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.996983] LNetError: 58039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.022661] LNet: Removed LNI 10.1.2.3@tcp [ 1028.052628] Key type .llcrypt unregistered [ 1028.057736] Key type ._llcrypt unregistered [ 1036.040322] Key type ._llcrypt registered [ 1036.041877] Key type .llcrypt registered [ 1036.288870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.314048] alg: No test for adler32 (adler32-zlib) [ 1037.266269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1037.293206] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1045.314114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1046.114889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1046.166465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1046.207170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1046.230763] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1046.486186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.497277] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1046.532388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.610638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.656692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.713442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.769861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.794898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.852920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.877941] LNet: 58481:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.931286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.983293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.032138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.070895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.109807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.178614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.225265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.844695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1047.888215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1047.923690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1047.939400] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1048.188171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.217830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.249174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.282395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.323501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.371905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.424618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.457676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.484571] LNet: 58590:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.509511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.601102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.643734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.686169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.736324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.772465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.831898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.371658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1049.406865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1049.442754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1049.468576] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1049.474331] LNet: Skipped 1 previous similar message [ 1049.718475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.779622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1049.849586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.908422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.995523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.049888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.089435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.124276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.142454] LNet: 58700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.161668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.197917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.229717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.263416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.297910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.338411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.384468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1050.961934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1050.999830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1051.043458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1051.319921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.348357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.375885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.405613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.437936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.468722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.507470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.537342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.595253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.633698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.682758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.748328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.796868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.838070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.874162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.519451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1052.556668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1052.602948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1052.646196] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1052.653423] LNet: Skipped 1 previous similar message [ 1052.861564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.894304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.936823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.971362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.001905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.045314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.074726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.104727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.122193] LNet: 58918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.127588] LNet: 58918:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1053.157294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.199027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.229756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.274728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.309772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.342890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.372194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.993443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1054.030947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1054.075497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1054.365126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.414415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.455216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.493603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.526572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.563431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.610657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.651227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.695936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.728593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.757989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.796506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.834743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.888607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.950917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1055.232596] LNet: 59073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.237956] LNetError: 59073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.298683] LNet: Removed LNI 192.168.204.40@tcp [ 1056.773214] Key type .llcrypt unregistered [ 1056.777546] Key type ._llcrypt unregistered [ 1074.239763] Key type ._llcrypt registered [ 1074.246282] Key type .llcrypt registered [ 1075.573604] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:31:08 (1787787068) [ 1076.348412] Key type .llcrypt unregistered [ 1076.350760] Key type ._llcrypt unregistered [ 1082.245202] Key type ._llcrypt registered [ 1082.247600] Key type .llcrypt registered [ 1082.425688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.440984] alg: No test for adler32 (adler32-zlib) [ 1083.356321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.397150] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1090.610256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1100.436178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.501488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1100.527947] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1101.828485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1101.860667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1101.908454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1101.953813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1101.998955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1102.040624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1102.085631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1102.147710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1102.178804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1102.194903] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1102.198879] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1102.378600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.399704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.422629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.445904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.467726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.488360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.509243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.533352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.548730] LNet: 60562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.563750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.597867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.636731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.670128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.707843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.740260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.768974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.291550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1103.371241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1103.404216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1103.434471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1103.463961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1103.488369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1103.513387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1103.540728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1103.563763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1103.584297] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1103.593880] LNet: Skipped 2 previous similar messages [ 1103.848773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.887964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.924868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.961918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.995502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.066650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.140470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.226365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.261618] LNet: 60683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.282038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.318745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.342877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.378158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.409659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.456673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.480649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.851532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1104.875981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1104.908463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1104.942731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1104.988428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1105.034646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1105.064382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1105.085908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1105.119762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1105.138396] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1105.141963] LNet: Skipped 2 previous similar messages [ 1105.347264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.384993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.407076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.434627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.453958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.491138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.523227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.565381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.577731] LNet: 60804:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.595923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.636451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.687318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.712131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.739622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.768202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.788316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.335270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1106.390958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1106.437925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1106.470420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1106.505865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1106.550945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1106.585706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1106.636624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1106.679932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1106.857481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.882994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.922906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.950706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.972728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.997424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.020820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.044848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.083883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.114746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.143930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.186123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.234744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.292396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.341253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.030928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1108.107150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1108.158797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1108.240911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1108.284258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1108.326848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1108.371919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1108.416419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1108.458687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1108.481823] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1108.501496] LNet: Skipped 5 previous similar messages [ 1108.715564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.751517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.774560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.796504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.820187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.843220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.874518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.902850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.916072] LNet: 61047:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.920766] LNet: 61047:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1108.935931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.963649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.002484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.036595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.076097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.107517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.147566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.697964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1109.738952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1109.770367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1109.817338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1109.859211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1109.898983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1109.951402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1110.008921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1110.065405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1110.306507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.351968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.393762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.428926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.493344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.564244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.597742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.625918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.655943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.687354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.710170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.730677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.752776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.776478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.798247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.035873] LNet: 61248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.042469] LNetError: 61248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.051169] LNet: Removed LNI 192.168.204.40@tcp [ 1116.067045] LNet: Removed LNI 192.168.204.40@tcp1 [ 1116.523682] Key type .llcrypt unregistered [ 1116.539221] Key type ._llcrypt unregistered [ 1132.926290] Key type ._llcrypt registered [ 1132.929707] Key type .llcrypt registered [ 1134.383542] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:32:06 (1787787126) [ 1135.267322] Key type .llcrypt unregistered [ 1135.268749] Key type ._llcrypt unregistered [ 1142.116935] Key type ._llcrypt registered [ 1142.119475] Key type .llcrypt registered [ 1142.416303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.431534] alg: No test for adler32 (adler32-zlib) [ 1143.493382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.531849] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1151.131956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1151.918498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1151.975193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1152.037914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1152.064724] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1152.303256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.335763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.372444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.412948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.437210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.464239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.495400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.529486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.561324] LNet: 62504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.597727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.632830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.680142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.727840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.762991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.812889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.878828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1153.516577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1153.568405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1153.600778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1153.619385] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1153.848739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.886971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.920913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.951279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.987626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.021974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.058790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.093720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.110339] LNet: 62613:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.129389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.158960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.191485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.222976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.268343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.309326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.352889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1154.932292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1154.963755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1154.993781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1155.008412] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1155.229286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.278249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.313170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.342646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.373187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.401149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.432937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.459423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.477948] LNet: 62722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.497650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.524314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.546890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.569966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.601505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.640234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.694578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.019811] LNet: 62767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.032561] LNetError: 62767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.056440] LNet: Removed LNI 192.168.204.40@tcp [ 1156.461353] Key type .llcrypt unregistered [ 1156.466089] Key type ._llcrypt unregistered [ 1170.578260] Key type ._llcrypt registered [ 1170.584172] Key type .llcrypt registered [ 1172.056807] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:32:44 (1787787164) [ 1173.186918] Key type .llcrypt unregistered [ 1173.190758] Key type ._llcrypt unregistered [ 1179.969372] Key type ._llcrypt registered [ 1179.974584] Key type .llcrypt registered [ 1180.329597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.349650] alg: No test for adler32 (adler32-zlib) [ 1181.400797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1181.440268] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1189.385984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1202.035170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1202.075573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1202.106856] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1203.344487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1203.376394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1203.401677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1203.449113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1203.483647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1203.518964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1203.553474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1203.596361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1203.643143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1203.678280] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1203.688512] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1203.912218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.938685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.971732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.004479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.037988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.072611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.117267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.166956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.186583] LNet: 64240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.212874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.248585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.293959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.355096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.402986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.458598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.491644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1204.510465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.968490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1205.032906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1205.101780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1205.162494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1205.201702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1205.256713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1205.324359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1205.386162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1205.430749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1205.449515] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1205.455391] LNet: Skipped 2 previous similar messages [ 1205.732441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.783658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.855450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.909318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.958140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.990213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.009809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.039553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.056569] LNet: 64363:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.077129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.117909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.148638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.173436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.204416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.240276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.282225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.324659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.782689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1206.817505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1206.849921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1206.879351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1206.908929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1206.962296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1207.010926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1207.066344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1207.106690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1207.171802] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1207.524274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.563630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.590105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.628741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.688862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.753506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.818952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.863509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.893046] LNet: 64486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.919148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.976136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.042249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.119341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.176947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.225592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.283289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.343660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.143162] LNet: 64569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.166594] LNetError: 64569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.175572] LNet: Removed LNI 192.168.204.40@tcp [ 1214.200728] LNet: Removed LNI 192.168.204.40@tcp1 [ 1214.625902] Key type .llcrypt unregistered [ 1214.627878] Key type ._llcrypt unregistered [ 1228.991693] Key type ._llcrypt registered [ 1228.993060] Key type .llcrypt registered [ 1230.019992] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:33:43 (1787787223) [ 1231.622209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1233.123822] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:33:46 (1787787226) [ 1234.071288] Key type .llcrypt unregistered [ 1234.075169] Key type ._llcrypt unregistered [ 1239.733971] Key type ._llcrypt registered [ 1239.737535] Key type .llcrypt registered [ 1239.913334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.935898] alg: No test for adler32 (adler32-zlib) [ 1240.831293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.861653] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1247.745505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1248.612763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1248.649742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1248.696508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1248.715726] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1248.914876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.961796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.996666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.026534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.057482] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1249.065333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.096765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.120380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.154553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.179802] LNet: 66116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.206323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.243355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.307726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.342614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.369837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.410042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.713655] LNet: 66159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.725166] LNetError: 66159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.744720] LNet: Removed LNI 192.168.204.40@tcp [ 1250.292886] Key type .llcrypt unregistered [ 1250.301697] Key type ._llcrypt unregistered [ 1269.240112] Key type ._llcrypt registered [ 1269.242710] Key type .llcrypt registered [ 1269.565916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.602204] alg: No test for adler32 (adler32-zlib) [ 1270.517293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.550934] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1277.435505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1287.169275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.230551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1287.267675] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1288.712037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1288.736945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1288.785162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1288.849210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1288.902455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1288.989410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1289.040874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1289.088478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1289.137726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1289.171290] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1289.457429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.487220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.518649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.567472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.640104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.696525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.753149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.827632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.878521] LNet: 67044:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1289.941432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.001929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.039359] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1290.042093] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1290.054788] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1290.096741] LNet: Skipped 1 previous similar message [ 1290.109422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.213587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.273353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.338775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.968898] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1295.540648] LNet: 67122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.548623] LNetError: 67122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.607389] LNet: Removed LNI 192.168.204.40@tcp [ 1296.610172] LNetError: 67122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.655427] LNet: Removed LNI 192.168.204.40@tcp1 [ 1297.161444] Key type .llcrypt unregistered [ 1297.170399] Key type ._llcrypt unregistered [ 1315.370591] Key type ._llcrypt registered [ 1315.373350] Key type .llcrypt registered [ 1316.772185] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:35:09 (1787787309) [ 1317.250539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1317.268618] alg: No test for adler32 (adler32-zlib) [ 1318.348930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.429609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1318.458940] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1319.123757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1319.146337] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1319.907233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1320.020485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1320.076645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1320.103360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1320.129250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1320.159712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1320.185914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1320.206268] LNetError: 68224:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1320.480611] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1320.480615] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1322.527401] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1323.623436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1326.625899] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1327.182389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1327.217914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1327.280065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1327.295945] LNet: 68332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.302178] LNetError: 68332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.351444] LNet: Removed LNI 192.168.204.40@tcp [ 1328.354738] LNetError: 68332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.370547] LNet: Removed LNI 192.168.204.40@tcp1 [ 1328.385557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.420992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.436624] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1329.035298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.051982] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1329.784205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1329.817872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.872608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1329.920950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1329.953331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1329.988424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1330.022648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1330.052879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1330.074649] LNetError: 68604:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1330.463776] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1333.474828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1336.607674] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1336.620447] LNet: Skipped 1 previous similar message [ 1341.266820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1341.303745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.346696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.442916] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:35:40 (1787787340) [ 1347.543767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.562450] LNet: 69313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.570305] LNetError: 69313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.574250] LNet: Removed LNI 192.168.204.40@tcp [ 1347.587889] LNet: Removed LNI 192.168.204.40@tcp1 [ 1347.616666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.685450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1347.702311] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1348.342675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.355786] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1348.945926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1349.027621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1349.073966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1349.132568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1349.168274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1349.201867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1349.222150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1349.297168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1349.728507] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.40@tcp [ 1350.751505] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1350.756535] LNet: Skipped 1 previous similar message [ 1364.497305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1364.634201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1364.662666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.712221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 500 [ 1364.815522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1370.321982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1370.387527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.423709] LNet: 69804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.431847] LNetError: 69804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.444033] LNetError: 69804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1370.452815] LNet: Removed LNI 192.168.204.40@tcp [ 1370.466299] LNet: Removed LNI 192.168.204.40@tcp1 [ 1370.488252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.553880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1370.578609] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1371.270581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1371.295194] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1372.220562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1372.248395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.308599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1372.349429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1372.404781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1372.442186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1372.486825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1372.551977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1373.606963] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1373.617041] LNet: Skipped 3 previous similar messages [ 1373.902321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.184576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.232712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1384.283325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.357161] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:36:24 (1787787384) [ 1392.102543] LNet: 70851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.108200] LNetError: 70851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.113729] LNetError: 70851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1392.119889] LNet: Removed LNI 192.168.204.40@tcp [ 1392.129516] LNet: Removed LNI 192.168.204.40@tcp1 [ 1392.875879] Key type .llcrypt unregistered [ 1392.881027] Key type ._llcrypt unregistered [ 1398.973374] Key type ._llcrypt registered [ 1398.976285] Key type .llcrypt registered [ 1399.366600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.401642] alg: No test for adler32 (adler32-zlib) [ 1400.311623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1400.336376] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1408.987403] Lustre: DEBUG MARKER: Initial discovery [ 1409.154527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1411.690459] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1411.828737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1411.876905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1411.901332] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1411.920165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1411.961652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -l 3 [ 1412.005226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -e local_error [ 1412.064549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m PUT -l 6 [ 1412.640646] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1413.573936] Lustre: DEBUG MARKER: Force oleg440-client.virtnet to discover 192.168.204.140@tcp (in background) [ 1413.961563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1415.338310] Lustre: DEBUG MARKER: Force oleg440-server to discover 192.168.204.40@tcp [ 1417.058944] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1417.902692] Lustre: DEBUG MARKER: Wait for 71620 [ 1419.693215] Lustre: DEBUG MARKER: Finished wait on 71620 [ 1419.812454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1419.851493] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1419.877728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1420.263275] LNet: 72072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.272597] LNetError: 72072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.305523] LNet: Removed LNI 192.168.204.40@tcp [ 1420.826913] Key type .llcrypt unregistered [ 1420.831219] Key type ._llcrypt unregistered [ 1438.795710] Key type ._llcrypt registered [ 1438.799444] Key type .llcrypt registered [ 1440.156952] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:37:12 (1787787432) [ 1440.764799] Key type .llcrypt unregistered [ 1440.766787] Key type ._llcrypt unregistered [ 1446.711565] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1446.873104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1446.877715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1446.880863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1449.536794] Key type ._llcrypt registered [ 1449.541959] Key type .llcrypt registered [ 1449.782386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1449.805750] alg: No test for adler32 (adler32-zlib) [ 1450.808620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.897482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.964956] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1451.850740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1451.873084] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1460.509914] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:37:33 (1787787453) [ 1460.899401] LNet: 74155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.915592] LNetError: 74155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1460.922885] LNet: Removed LNI 10.1.2.3@tcp [ 1460.940738] LNet: Removed LNI 192.168.204.40@tcp [ 1461.592560] Key type .llcrypt unregistered [ 1461.597575] Key type ._llcrypt unregistered [ 1468.363215] Key type ._llcrypt registered [ 1468.364549] Key type .llcrypt registered [ 1468.528127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1468.542461] alg: No test for adler32 (adler32-zlib) [ 1469.596721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.663809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.716936] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1470.500269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1470.524851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1471.403303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.40@tcp [ 1472.479269] LNetError: Unexpected error -22 connecting to 192.168.204.40@tcp at host 192.168.204.40:988 [ 1477.907925] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:37:50 (1787787470) [ 1478.024199] LNet: 75283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.029965] LNetError: 75283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1478.035262] LNet: Removed LNI 10.1.2.3@tcp [ 1478.092444] LNet: Removed LNI 192.168.204.40@tcp [ 1479.090816] Key type .llcrypt unregistered [ 1479.095253] Key type ._llcrypt unregistered [ 1485.308635] Key type ._llcrypt registered [ 1485.310949] Key type .llcrypt registered [ 1485.477478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1485.493977] alg: No test for adler32 (adler32-zlib) [ 1486.465419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.514766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.530859] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1487.262276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1487.297359] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1488.064195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.40@tcp1 --nid 192.168.204.40@tcp2 [ 1491.867535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp2 [ 1491.883427] LNetError: 76132:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.40@tcp1 to 192.168.204.40@tcp2 - Route table may be misconfigured [ 1491.903226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp2 192.168.204.40@tcp1 [ 1498.201184] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:38:10 (1787787490) [ 1498.356478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.377520] LNet: 76704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.384135] LNetError: 76704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.426394] LNet: Removed LNI 192.168.204.40@tcp1 [ 1499.431061] LNetError: 76704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.456964] LNet: Removed LNI 192.168.204.40@tcp2 [ 1499.510243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.554468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.586636] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1500.368577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1500.390736] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1501.121877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1501.150573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.40@tcp [ 1501.178582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp1 [ 1501.208968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp -e network_timeout [ 1501.233671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -e network_timeout [ 1501.257935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -e network_timeout [ 1501.289794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -e network_timeout [ 1501.337976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1501.367611] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1501.388136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1501.473711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.557934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1501.600487] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1507.162878] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:38:20 (1787787500) [ 1507.396533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.422288] LNet: 77566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.430472] LNetError: 77566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.436854] LNet: Removed LNI 192.168.204.40@tcp [ 1508.453344] LNet: Removed LNI 192.168.204.40@tcp1 [ 1508.487715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.572284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1508.901710] LNet: 77607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.283409] Key type .llcrypt unregistered [ 1509.286344] Key type ._llcrypt unregistered [ 1519.041517] Key type ._llcrypt registered [ 1519.048536] Key type .llcrypt registered [ 1520.313670] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:38:33 (1787787513) [ 1520.702314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1520.714178] alg: No test for adler32 (adler32-zlib) [ 1521.634542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.809476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1524.343340] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:38:37 (1787787517) [ 1524.482555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.499493] LNet: 78704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.522123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.591235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.640707] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1525.384514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1525.410086] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1526.269229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1526.306415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp1 [ 1526.344428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp1 [ 1531.729523] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:38:44 (1787787524) [ 1533.533691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg440-server [ 1535.069723] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:38:48 (1787787528) [ 1536.747272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg440-server [ 1538.333971] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:38:51 (1787787531) [ 1540.078444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg440-server [ 1541.739475] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:38:54 (1787787534) [ 1543.567849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg440-server [ 1545.118358] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:38:58 (1787787538) [ 1546.831425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg440-server [ 1548.335590] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:39:01 (1787787541) [ 1549.969267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg440-server [ 1551.613205] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:39:04 (1787787544) [ 1553.305057] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg440-server [ 1555.196969] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:39:07 (1787787547) [ 1557.304250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg440-server [ 1559.138671] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:39:11 (1787787551) [ 1561.248501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg440-server [ 1563.044646] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:39:15 (1787787555) [ 1563.370645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.422371] LNet: 82581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.436393] LNetError: 82581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.440269] LNet: Removed LNI 192.168.204.40@tcp [ 1564.460418] LNet: Removed LNI 192.168.204.40@tcp1 [ 1564.486831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.555761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.585549] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1565.300823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1565.395807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1565.525176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.546170] LNet: 82729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.550900] LNetError: 82729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.556498] LNetError: 82729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1565.578148] LNet: Removed LNI 192.168.204.40@tcp [ 1565.606750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.647633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1565.668917] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1566.292850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1566.444365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.534644] LNet: Removed LNI 192.168.204.40@tcp [ 1567.558241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.600439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.625818] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1568.382746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1568.459078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1568.655477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.685963] LNet: 83021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.695123] LNet: 83021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1568.712645] LNetError: 83021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.723493] LNetError: 83021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1568.752538] LNet: Removed LNI 192.168.204.40@tcp [ 1568.811260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.902600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1568.928907] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1569.696227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1569.978231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.048387] LNet: Removed LNI 192.168.204.40@tcp [ 1571.075223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.173793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.221549] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1572.167401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1572.282818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1572.685314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.747878] LNet: 83314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.756869] LNet: 83314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1572.767288] LNetError: 83314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.777025] LNetError: 83314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1572.813749] LNet: Removed LNI 192.168.204.40@tcp [ 1572.865555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.914353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1572.961203] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1573.753868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1573.994941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.077873] LNet: Removed LNI 192.168.204.40@tcp [ 1575.099222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.142952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.162409] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1575.986509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1576.089709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1577.543342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.570611] LNetError: 83616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.578642] LNetError: 83616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1577.617165] LNet: Removed LNI 192.168.204.40@tcp [ 1577.670797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.759895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1577.807358] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1578.566148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1578.888846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.958943] LNet: Removed LNI 192.168.204.40@tcp [ 1579.991688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.032415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.049645] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1580.791471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1580.910714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1581.182846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.197892] LNet: 83908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.201846] LNet: 83908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1582.265641] LNet: Removed LNI 192.168.204.40@tcp [ 1582.299607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.337598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1582.350414] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1582.923138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1584.307766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.386301] LNet: Removed LNI 192.168.204.40@tcp [ 1585.433828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.512293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.552976] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1586.506791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1586.629832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1586.861326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.882924] LNetError: 84210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.887138] LNetError: 84210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1586.903663] LNet: Removed LNI 192.168.204.40@tcp [ 1586.933281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.977654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1586.997910] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1587.595743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1587.908153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.986862] LNet: Removed LNI 192.168.204.40@tcp [ 1589.025595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.136488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.171355] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1590.082086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1590.135769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.186699] LNet: Removed LNI 192.168.204.40@tcp [ 1590.237076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.365402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.413280] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1591.258463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1591.319809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.344780] LNet: Removed LNI 192.168.204.40@tcp [ 1591.361702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.424788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1591.493917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.522932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.556181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1591.673707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.733595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.776553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1591.880058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.932419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.001387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1592.052558] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1593.043134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.081758] LNet: Removed LNI 192.168.204.40@tcp [ 1593.104435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.196754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1599.854365] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:39:52 (1787787592) [ 1600.080924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.121570] LNet: 85377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.140363] LNet: 85377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1600.206289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.273240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1600.304573] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1601.566529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.605714] LNet: Removed LNI 192.168.204.40@tcp231 [ 1601.640859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.718144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1601.768260] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1601.774870] LNet: 85523:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1602.152293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1602.184860] LNet: Removed LNI 192.168.204.40@tcp231 [ 1602.205825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1602.230787] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1603.396325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.428300] LNetError: 85668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.439215] LNetError: 85668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1603.461857] LNet: Removed LNI 192.168.204.40@tcp231 [ 1603.496435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.594442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1603.640553] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1603.649245] LNet: 85676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1604.098389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1604.136359] LNet: Removed LNI 192.168.204.40@tcp231 [ 1604.180123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1604.235453] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1605.365785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.388764] LNet: Removed LNI 192.168.204.40@tcp231 [ 1605.414067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.528684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1605.566588] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1605.575280] LNet: 85827:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1606.119942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1606.157367] LNet: Removed LNI 192.168.204.40@tcp231 [ 1606.185312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1606.206476] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1607.241661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.284688] LNet: Removed LNI 192.168.204.40@tcp231 [ 1607.314483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.381236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1607.414537] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1608.043230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1608.102812] LNet: Removed LNI 192.168.204.40@tcp231 [ 1608.128251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1608.176266] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1616.806139] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:40:09 (1787787609) [ 1616.959802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.985874] LNet: Removed LNI 192.168.204.40@tcp231 [ 1617.011638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.111387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1617.148441] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1617.918688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1617.954558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1617.987903] LNet: Removed LNI 192.168.204.40@tcp232 [ 1618.000478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1618.020838] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1618.907080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1618.967777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1619.025295] LNet: Removed LNI 192.168.204.40@tcp232 [ 1619.044832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1619.068299] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1619.800362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1619.857717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1619.909053] LNet: Removed LNI 192.168.204.40@tcp232 [ 1619.938934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1619.971989] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1620.886646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1620.920930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1620.952207] LNet: Removed LNI 192.168.204.40@tcp232 [ 1620.977405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1627.894113] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:40:20 (1787787620) [ 1628.915428] Key type .llcrypt unregistered [ 1628.919585] Key type ._llcrypt unregistered [ 1636.002561] Key type ._llcrypt registered [ 1636.005124] Key type .llcrypt registered [ 1636.283399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1636.293640] alg: No test for adler32 (adler32-zlib) [ 1637.227311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1637.260028] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1644.713762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1654.226472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.267942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1654.292841] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1655.561201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1655.592349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1655.630206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1655.662379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1655.692979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1655.721996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1655.773762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1655.811875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1655.844700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1655.875426] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1655.891592] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1656.131972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1656.187387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1656.230938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1656.274890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1656.319504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1656.389484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1656.479546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1656.530788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1656.556058] LNet: 88486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1656.579969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.657525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.696418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.729829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.764709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.801927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1657.621660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1657.662767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1657.706777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1657.776370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1657.833232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1657.873232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1657.959029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1658.004760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1658.066520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1658.105895] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1658.509701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1658.591179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1658.674455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1658.761726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1658.812545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1658.858015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1658.895656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.943738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1658.966703] LNet: 88606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.010286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1659.063315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1659.107870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1659.175143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1659.223438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1659.279867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1659.947744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1660.019269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1660.054914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1660.082416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1660.113111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1660.145601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1660.198953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1660.248665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1660.295063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1660.343915] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1660.699724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1660.741996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1660.776988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1660.826600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1660.858303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1660.901221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1660.949139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.002760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1661.032620] LNet: 88727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.055368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1661.099323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1661.128468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1661.207189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1661.248490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1661.298318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.921952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1661.972268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1662.011486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1662.087528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1662.158963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1662.232903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1662.273960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1662.318421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1662.356442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1662.389379] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1662.778908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1662.818968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1662.847766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1662.879496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1662.957488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1663.024605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1663.067541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1663.111923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1663.131964] LNet: 88847:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.158668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1663.184743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1663.225402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1663.283356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1663.336118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1663.389427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.979885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1664.017494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1664.049537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1664.089993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1664.152239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1664.262446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1664.333831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1664.367927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1664.399865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1664.658712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1664.720181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1664.815913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1664.883020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1664.937273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1664.987291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1665.043786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1665.088579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1665.130526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1665.184237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1665.259096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1665.317349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1665.371195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1665.433661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1666.276490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1666.345615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1666.386927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1666.433976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1666.467380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1666.499321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1666.528164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1666.564242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1666.606769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1666.632662] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1666.638344] LNet: Skipped 1 previous similar message [ 1666.641352] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1666.824486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1666.849644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1666.879562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1666.924232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1666.986667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1667.015878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1667.044241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1667.085205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1667.127115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1667.163994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1667.194877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1667.231183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1667.267542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1667.301911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1673.163186] LNet: 89166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.174514] LNetError: 89166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.181508] LNet: Removed LNI 192.168.204.40@tcp [ 1673.196961] LNet: Removed LNI 192.168.204.40@tcp1 [ 1673.644606] Key type .llcrypt unregistered [ 1673.649365] Key type ._llcrypt unregistered [ 1694.676147] Key type ._llcrypt registered [ 1694.680501] Key type .llcrypt registered [ 1695.965314] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:41:28 (1787787688) [ 1696.609967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.651719] alg: No test for adler32 (adler32-zlib) [ 1697.579814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.678057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.702828] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1698.432811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1698.465924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1698.485672] LNetError: 90146:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1698.582921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.40@tcp [ 1698.697358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1705.344133] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:41:37 (1787787697) [ 1705.444400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.465833] LNet: 90728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.471746] LNetError: 90728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.497461] LNet: Removed LNI 192.168.204.40@tcp [ 1705.516105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.588822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.621372] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1706.607043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1706.650781] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1707.783369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.40@tcp [ 1707.803143] LNetError: 91002:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1708.639357] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1708.652277] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1709.039878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.40@tcp [ 1717.666481] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:41:50 (1787787710) [ 1719.372696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg440-server [ 1721.201572] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:41:53 (1787787713) [ 1723.349766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg440-server [ 1725.175347] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:41:57 (1787787717) [ 1726.788927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg440-server [ 1728.296628] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:42:01 (1787787721) [ 1728.776991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.793713] LNet: 92632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.801979] LNetError: 92632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.813858] LNet: Removed LNI 192.168.204.40@tcp [ 1728.847961] LNet: Removed LNI 192.168.204.40@tcp1 [ 1728.867840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.944929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.962269] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1729.723850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1729.795188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1729.833937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1729.913088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1729.951807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1729.979712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1738.024400] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:42:10 (1787787730) [ 1738.157382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.220107] LNet: 93350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.235399] LNetError: 93350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.249332] LNetError: 93350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1738.292137] LNet: Removed LNI 192.168.204.40@tcp [ 1738.365413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.475553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.506384] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1746.359501] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:42:19 (1787787739) [ 1746.959254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.000879] LNet: 94060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.011124] LNetError: 94060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.045245] LNet: Removed LNI 192.168.204.40@tcp [ 1747.067408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.143649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.179861] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1755.750476] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:42:28 (1787787748) [ 1757.445678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1759.386637] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:42:32 (1787787752) [ 1760.074292] LNet: 95095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.088373] LNetError: 95095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.110246] LNet: Removed LNI 192.168.204.40@tcp [ 1760.678541] Key type .llcrypt unregistered [ 1760.681125] Key type ._llcrypt unregistered [ 1767.412253] Key type ._llcrypt registered [ 1767.415467] Key type .llcrypt registered [ 1767.672470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1767.736772] alg: No test for adler32 (adler32-zlib) [ 1768.742113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1768.788910] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1777.439962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1782.626732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.140@tcp [ 1783.091472] LNet: 95522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.097549] LNetError: 95522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.117780] LNet: Removed LNI 192.168.204.40@tcp [ 1783.649987] Key type .llcrypt unregistered [ 1783.655869] Key type ._llcrypt unregistered [ 1796.344850] Key type ._llcrypt registered [ 1796.348280] Key type .llcrypt registered [ 1797.776925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1799.393361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1801.234451] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:43:14 (1787787794) [ 1802.977126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1804.639776] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:43:17 (1787787797) [ 1806.422220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg440-server [ 1808.183394] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:43:21 (1787787801) [ 1809.936579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg440-server [ 1811.653303] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:43:24 (1787787804) [ 1813.026606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1813.120160] alg: No test for adler32 (adler32-zlib) [ 1814.214457] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 1814.491478] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1816.175166] Key type lgssc registered [ 1817.460573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1837.750406] Key type lgssc unregistered [ 1838.005399] LNet: 98714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.016502] LNetError: 98714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.028572] LNet: Removed LNI 192.168.204.40@tcp [ 1838.531877] Key type .llcrypt unregistered [ 1838.534681] Key type ._llcrypt unregistered [ 1853.723862] Key type ._llcrypt registered [ 1853.725975] Key type .llcrypt registered [ 1855.615722] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:44:07 (1787787847) [ 1856.134679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.150259] alg: No test for adler32 (adler32-zlib) [ 1857.049872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.109875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1857.178359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1857.242549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1857.278759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1864.783346] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:44:17 (1787787857) [ 1865.146143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.176938] LNet: 100129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.194797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.223854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.239179] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1866.088454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.104319] LNet: 100274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.108271] LNetError: 100274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.125462] LNet: Removed LNI 192.168.204.40@tcp [ 1866.162489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.224565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1866.250571] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1866.904330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.924780] LNetError: 100302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.939447] LNet: Removed LNI 192.168.204.40@tcp [ 1866.951904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.990803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1867.015988] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1873.455843] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:44:26 (1787787866) [ 1873.611499] LNet: 100888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.618913] LNet: 100888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1873.648129] LNetError: 100888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.676099] LNet: Removed LNI 192.168.204.40@tcp [ 1874.619696] Key type .llcrypt unregistered [ 1874.624743] Key type ._llcrypt unregistered [ 1881.671511] Key type ._llcrypt registered [ 1881.676310] Key type .llcrypt registered [ 1881.910390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1881.929709] alg: No test for adler32 (adler32-zlib) [ 1882.970790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.109963] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1883.113412] LNet: *** cfs_fail_loc=e003, val=100*** [ 1884.233804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1884.269189] LNet: *** cfs_fail_loc=e003, val=100*** [ 1884.273234] LNet: Skipped 2 previous similar messages [ 1884.302174] Ping reply received with mapping: [ 1884.309568] found metadata reply NI[192.168.204.40@tcp] [ 1884.317450] 28 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1884.327700] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1884.338016] 5a 5a 5a 5a 5a 5a 5a 5a [ 1884.498930] LNet: 101294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.513371] LNetError: 101294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.551371] LNet: Removed LNI 192.168.204.40@tcp [ 1885.478401] Key type .llcrypt unregistered [ 1885.482444] Key type ._llcrypt unregistered [ 1899.386439] Key type ._llcrypt registered [ 1899.393504] Key type .llcrypt registered [ 1900.653530] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:44:53 (1787787893) [ 1901.160576] Key type .llcrypt unregistered [ 1901.162058] Key type ._llcrypt unregistered [ 1907.062554] Key type ._llcrypt registered [ 1907.065906] Key type .llcrypt registered [ 1907.248291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.273475] alg: No test for adler32 (adler32-zlib) [ 1908.482189] LNetError: 102352:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1908.911451] Key type .llcrypt unregistered [ 1908.916168] Key type ._llcrypt unregistered [ 1915.342091] Key type ._llcrypt registered [ 1915.344515] Key type .llcrypt registered [ 1916.925613] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:45:09 (1787787909) [ 1917.599314] Key type .llcrypt unregistered [ 1917.601747] Key type ._llcrypt unregistered [ 1924.573557] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1924.580513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1927.290276] Key type ._llcrypt registered [ 1927.293037] Key type .llcrypt registered [ 1927.587604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.617398] alg: No test for adler32 (adler32-zlib) [ 1928.595930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.687944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1938.154969] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:45:30 (1787787930) [ 1938.521427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.564538] LNet: 104141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.576895] LNetError: 104141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1938.599870] LNet: Removed LNI 10.1.2.3@tcp [ 1938.656332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.785751] Key type .llcrypt unregistered [ 1939.787585] Key type ._llcrypt unregistered [ 1947.011906] Key type ._llcrypt registered [ 1947.014432] Key type .llcrypt registered [ 1947.196864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1947.216687] alg: No test for adler32 (adler32-zlib) [ 1948.130676] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1948.190837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1948.324294] LNet: 104432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.342191] LNetError: 104432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.364228] LNet: Removed LNI 192.168.204.40@tcp [ 1948.935444] Key type .llcrypt unregistered [ 1948.938907] Key type ._llcrypt unregistered [ 1961.542700] Key type ._llcrypt registered [ 1961.545120] Key type .llcrypt registered [ 1962.898773] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:45:55 (1787787955) [ 1963.638421] Key type .llcrypt unregistered [ 1963.643096] Key type ._llcrypt unregistered [ 1970.814718] Key type ._llcrypt registered [ 1970.823022] Key type .llcrypt registered [ 1971.020695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1971.038541] alg: No test for adler32 (adler32-zlib) [ 1979.142693] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:46:11 (1787787971) [ 1979.504488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1979.541455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.631961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.657560] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1980.475713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.527744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1980.570851] LNet: Removed LNI 192.168.204.40@tcp [ 1980.946126] LNet: 106188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.413486] Key type .llcrypt unregistered [ 1981.422067] Key type ._llcrypt unregistered [ 1994.146772] Key type ._llcrypt registered [ 1994.148901] Key type .llcrypt registered [ 1995.676785] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:46:28 (1787787988) [ 1996.133572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.150470] alg: No test for adler32 (adler32-zlib) [ 1997.116859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.218560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.244585] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1998.169436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1998.234985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.40@tcp [ 1998.268641] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1998.272932] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.277149] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 0 [ 1998.283061] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1998.285084] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1998.287764] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.290201] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 0 [ 1998.294231] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1998.296727] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1998.299829] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.302105] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 2048 [ 1998.305228] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1998.306706] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1998.309977] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.313210] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 2048 [ 1998.323138] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2005.797330] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:46:38 (1787787998) [ 2006.345567] LNet: 107729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.364934] LNetError: 107729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.397975] LNet: Removed LNI 192.168.204.40@tcp [ 2006.964709] Key type .llcrypt unregistered [ 2006.968956] Key type ._llcrypt unregistered [ 2013.712904] Key type ._llcrypt registered [ 2013.714921] Key type .llcrypt registered [ 2013.933578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.952664] alg: No test for adler32 (adler32-zlib) [ 2014.886749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2014.950515] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2022.916164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2032.523610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.595510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2032.637806] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 2033.764427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2034.028438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2034.038930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2034.045130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2036.467102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2036.509473] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2049.040465] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:47:21 (1787788041) [ 2049.253696] LNet: 109234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.263605] LNetError: 109234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.282419] LNet: Removed LNI 192.168.204.40@tcp [ 2049.297242] LNet: Removed LNI 192.168.204.40@tcp1 [ 2049.334690] LNet: Removed LNI 10.1.2.3@tcp99 [ 2050.447439] Key type .llcrypt unregistered [ 2050.451727] Key type ._llcrypt unregistered [ 2060.608647] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2060.612985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2063.321917] Key type ._llcrypt registered [ 2063.323917] Key type .llcrypt registered [ 2063.500895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.526729] alg: No test for adler32 (adler32-zlib) [ 2064.563880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.661175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.703849] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2065.438466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2065.459751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2068.326657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2068.381927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2068.464815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2068.525196] LNetError: 109990:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2068.547198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2068.595626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2068.956882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2069.065499] LNet: 110010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.073688] LNetError: 110010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2069.080500] LNet: Removed LNI 10.1.2.3@tcp [ 2070.119951] LNet: Removed LNI 192.168.204.40@tcp [ 2070.893531] Key type .llcrypt unregistered [ 2070.896639] Key type ._llcrypt unregistered [ 2083.577704] Key type ._llcrypt registered [ 2083.579798] Key type .llcrypt registered [ 2084.880901] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:47:57 (1787788077) [ 2085.419460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2085.438066] alg: No test for adler32 (adler32-zlib) [ 2086.375750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.447208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.476390] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2087.276498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2087.376410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg440-client@tcp [ 2096.189454] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:48:08 (1787788088) [ 2096.281201] LNet: 111539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.286785] LNetError: 111539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.304506] LNet: Removed LNI 192.168.204.40@tcp [ 2097.257856] Key type .llcrypt unregistered [ 2097.265520] Key type ._llcrypt unregistered [ 2105.474912] Key type ._llcrypt registered [ 2105.477820] Key type .llcrypt registered [ 2105.718321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.732243] alg: No test for adler32 (adler32-zlib) [ 2106.707751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2106.746107] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2106.813280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2115.065083] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:48:27 (1787788107) [ 2117.180806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2119.128656] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:48:31 (1787788111) [ 2120.563643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2123.126391] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:48:35 (1787788115) [ 2123.267650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.306933] LNet: 112966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.320639] LNetError: 112966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.342704] LNet: Removed LNI 192.168.204.40@tcp [ 2123.373770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.453691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.491882] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2124.491326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2124.565499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2132.783827] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:48:45 (1787788125) [ 2132.985867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.047747] LNet: 113676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.077695] LNetError: 113676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.122957] LNet: Removed LNI 192.168.204.40@tcp [ 2133.201808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.295113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2133.352022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2134.077776] Key type .llcrypt unregistered [ 2134.082118] Key type ._llcrypt unregistered [ 2146.843167] Key type ._llcrypt registered [ 2146.846816] Key type .llcrypt registered [ 2148.041625] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:49:00 (1787788140) [ 2148.897228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.927427] alg: No test for adler32 (adler32-zlib) [ 2149.937882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.004762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.041338] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2150.957603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2151.061989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2158.945745] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:49:10 (1787788150) [ 2159.270490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.305280] LNet: 115243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.317382] LNetError: 115243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.336763] LNet: Removed LNI 192.168.204.40@tcp [ 2159.374712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.477492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2159.516650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2168.146692] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:49:20 (1787788160) [ 2168.276110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.310076] LNet: 115817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.356445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.470929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2169.134557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2169.216264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2169.323868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2170.021838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2170.089561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2170.212553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.230599] LNet: 115856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.253110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.361469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2170.721702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2170.761575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2170.836673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2171.340260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2171.369613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2179.271296] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:49:31 (1787788171) [ 2179.457185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.486717] LNet: 116451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.517766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.590389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2179.628402] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2180.674442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2180.695989] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2181.709674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2181.820318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2182.110187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.134783] LNetError: 116736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.145979] LNet: Removed LNI 192.168.204.40@tcp [ 2182.158888] LNet: Removed LNI 192.168.204.40@tcp2 [ 2182.190416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.243695] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2182.249117] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2182.254191] LNet: 116742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2182.604735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2182.913160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.967445] LNet: Removed LNI 192.168.204.40@tcp [ 2183.969224] LNetError: 116779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.974256] LNetError: 116779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2183.985616] LNet: Removed LNI 192.168.204.40@tcp2 [ 2184.004149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.058141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.078142] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2184.886631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2184.921377] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2185.990831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2186.056933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2186.167721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2186.372777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.398478] LNet: 117064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.405927] LNet: 117064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2186.412421] LNetError: 117064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.419783] LNet: Removed LNI 192.168.204.40@tcp [ 2187.495730] LNet: Removed LNI 192.168.204.40@tcp2 [ 2187.531463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.596742] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2187.601318] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2187.607795] LNet: 117070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2188.170857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2188.427613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.479372] LNet: Removed LNI 192.168.204.40@tcp [ 2189.542305] LNet: Removed LNI 192.168.204.40@tcp2 [ 2189.582517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.657459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.696064] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2190.575944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2190.604977] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2191.708314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2191.814892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2192.164659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.192919] LNetError: 117387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.202256] LNetError: 117387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2192.212844] LNet: Removed LNI 192.168.204.40@tcp [ 2192.221543] LNet: Removed LNI 192.168.204.40@tcp2 [ 2192.240455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.309990] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2192.316550] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2192.321280] LNet: 117393:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2192.742000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2192.937535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.016028] LNet: Removed LNI 192.168.204.40@tcp [ 2194.035971] LNet: Removed LNI 192.168.204.40@tcp2 [ 2194.115221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.191508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.212841] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2194.946510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2194.966580] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2195.871709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2195.957030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2196.026201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2196.420666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.452465] LNet: 117714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.459651] LNet: 117714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2197.535165] LNet: Removed LNI 192.168.204.40@tcp [ 2198.573195] LNet: Removed LNI 192.168.204.40@tcp2 [ 2198.620724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.714546] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2198.726102] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2198.737066] LNet: 117721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.214104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2199.451907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.468445] LNet: Removed LNI 192.168.204.40@tcp [ 2200.487850] LNet: Removed LNI 192.168.204.40@tcp2 [ 2200.529766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.684692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2200.724802] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2201.542947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2201.557937] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2202.453942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2202.484688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2202.560498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2202.972844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.995457] LNetError: 118054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.003649] LNetError: 118054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2203.014233] LNet: Removed LNI 192.168.204.40@tcp [ 2203.035216] LNet: Removed LNI 192.168.204.40@tcp2 [ 2203.055306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.105181] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2203.109791] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2203.113270] LNet: 118060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.720964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2204.005440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.087172] LNet: Removed LNI 192.168.204.40@tcp [ 2205.112303] LNet: Removed LNI 192.168.204.40@tcp2 [ 2205.172956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.231848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.252572] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2205.968840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2205.983336] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2206.863373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2206.925370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2206.962322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2207.005271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2207.401973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.415827] LNet: Removed LNI 192.168.204.40@tcp [ 2209.447733] LNet: Removed LNI 192.168.204.40@tcp2 [ 2209.504309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.601154] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2209.614985] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2210.281209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2219.346757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2221.055787] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:50:13 (1787788213) [ 2221.244293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.274049] LNet: 119180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.278828] LNet: 119180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2221.287330] LNetError: 119180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.297478] LNetError: 119180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2221.304523] LNet: Removed LNI 192.168.204.40@tcp [ 2222.380928] LNet: Removed LNI 192.168.204.40@tcp2 [ 2222.428163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.786603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.807837] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2223.852167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2223.902814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2225.121974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.147698] LNet: Removed LNI 10.1.2.3@tcp [ 2225.159732] LNet: Removed LNI 192.168.204.40@tcp [ 2225.197484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.266982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2225.284905] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2225.287700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2226.097259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2226.136338] LNet: Removed LNI 192.168.204.40@tcp [ 2226.162685] LNet: Removed LNI 10.1.2.3@tcp [ 2234.809469] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:50:27 (1787788227) [ 2234.906211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.968355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.041986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.073461] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2235.641757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2235.689523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.749186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.797819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.851943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.896569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.928904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2235.969159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2236.013428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2236.048474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2236.086414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2236.295550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2244.585980] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:50:36 (1787788236) [ 2244.724481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.801238] LNet: Removed LNI 192.168.204.40@tcp [ 2245.853403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.052900] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2246.148243] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2246.151916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2246.155247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2247.499641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2247.525925] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2248.548767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2248.592575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.275709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2251.342030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2251.452450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.40@tcp -m PUT -e local_timeout -r 1 [ 2251.530460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2251.622719] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2251.626123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2251.868559] LNetError: 120849:0:(lib-move.c:4801:lnet_parse()) 192.168.204.40@tcp, src 192.168.204.40@tcp: Dropping PUT (error -108 looking up sender) [ 2252.904545] LNet: Removed LNI 10.1.2.3@tcp [ 2253.929350] LNet: Removed LNI 192.168.204.40@tcp [ 2254.444314] Key type .llcrypt unregistered [ 2254.447229] Key type ._llcrypt unregistered [ 2266.073392] Key type ._llcrypt registered [ 2266.076194] Key type .llcrypt registered [ 2267.129446] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:51:00 (1787788260) [ 2267.572764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.585416] alg: No test for adler32 (adler32-zlib) [ 2268.490268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.874463] LNet: 122165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.412217] Key type .llcrypt unregistered [ 2269.417072] Key type ._llcrypt unregistered [ 2275.505207] Key type ._llcrypt registered [ 2275.509277] Key type .llcrypt registered [ 2275.854684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2275.871657] alg: No test for adler32 (adler32-zlib) [ 2276.820503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2276.847674] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2284.733802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2285.042644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2285.773559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 2286.057478] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 2286.099373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.137225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.179558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.246787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.308875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.360310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.427553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.505757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.559997] LNet: 122619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.596100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.665640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.705221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2286.743294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2286.789366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2286.846771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.601300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2287.945350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.009602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.081921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.109139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.136820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.165236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.202310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.245803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.275927] LNet: 122736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.308370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.352820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.390738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.430837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.474369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.516151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.293716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 2289.733552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.764385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2289.794075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.823381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.851796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.890745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.934371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.990195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2290.010853] LNet: 122853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2290.036279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2290.091568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2290.135584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2290.172364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.206265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.240822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.933775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2291.276462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.312424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2291.363962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.433485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.490588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.552416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.605925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.655638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.706395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.761939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.820102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.890828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.953829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2292.037714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2292.572390] LNet: 123020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.580576] LNetError: 123020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.605319] LNet: Removed LNI 192.168.204.40@tcp [ 2293.399973] Key type .llcrypt unregistered [ 2293.401532] Key type ._llcrypt unregistered [ 2312.813791] Key type ._llcrypt registered [ 2312.815832] Key type .llcrypt registered [ 2314.767733] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:51:46 (1787788306) [ 2315.162374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.178413] alg: No test for adler32 (adler32-zlib) [ 2316.207247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.686718] LNet: 123910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.320543] Key type .llcrypt unregistered [ 2317.326443] Key type ._llcrypt unregistered [ 2324.800178] Key type ._llcrypt registered [ 2324.802461] Key type .llcrypt registered [ 2324.952371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2324.962753] alg: No test for adler32 (adler32-zlib) [ 2325.863300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2325.896158] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2334.547936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2346.696647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.760547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2346.807030] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 2348.146420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2349.087724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 2349.476202] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 2349.499172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2349.560383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2349.614426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2349.673062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2349.737211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2349.777837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2349.834948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2349.881574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2349.931219] LNet: 124566:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2349.972271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2350.038650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2350.162873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2350.230262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2350.304145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2350.375840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.410026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2351.866804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2351.904752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2351.949769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2351.990506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2352.031720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2352.077884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2352.132439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2352.241906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2352.267697] LNet: 124683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.285336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2352.343192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2352.398152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2352.446226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2352.497362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2352.530508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.857518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 2354.280752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.346994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2354.410948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.464279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.539091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.610249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.710876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.782200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.813770] LNet: 124800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.858784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2354.939540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.025985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.109829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.178965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.219616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.514294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2356.969919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.046840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.142594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.226139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.321883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.381364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2357.463512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.554820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.594834] LNet: 124918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.648243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.773183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.882699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.944452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.014473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.143519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.438371] LNet: 125003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.454036] LNetError: 125003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.463108] LNet: Removed LNI 192.168.204.40@tcp [ 2365.494382] LNet: Removed LNI 192.168.204.40@tcp1 [ 2366.174533] Key type .llcrypt unregistered [ 2366.180296] Key type ._llcrypt unregistered [ 2386.808984] Key type ._llcrypt registered [ 2386.814765] Key type .llcrypt registered [ 2388.185252] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:53:00 (1787788380) [ 2389.028814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2389.055547] alg: No test for adler32 (adler32-zlib) [ 2390.031946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.132860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.153799] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2391.142985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2391.181706] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2392.234491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2392.302607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2393.375273] LNet: Removed LNI 192.168.204.40@tcp [ 2401.052261] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:53:13 (1787788393) [ 2402.850693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2404.735163] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:53:17 (1787788397) [ 2414.204984] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:53:26 (1787788406) [ 2422.192728] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:53:34 (1787788414) [ 2429.375515] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:53:42 (1787788422) [ 2437.942891] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:53:50 (1787788430) [ 2446.320154] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:53:58 (1787788438) [ 2454.471912] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:54:07 (1787788447) [ 2462.975544] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:54:15 (1787788455) [ 2470.483460] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:54:23 (1787788463) [ 2477.235352] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:54:30 (1787788470) [ 2484.505713] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:54:37 (1787788477) [ 2491.985802] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:54:44 (1787788484) [ 2499.255420] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:54:51 (1787788491) [ 2507.507413] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:55:00 (1787788500) [ 2515.625595] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:55:08 (1787788508) [ 2523.493746] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:55:16 (1787788516) [ 2531.546614] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:55:24 (1787788524) [ 2538.230940] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:55:31 (1787788531) [ 2538.305358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2538.323702] LNet: 136696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.337611] LNetError: 136696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.358200] LNet: Removed LNI 192.168.204.40@tcp2 [ 2538.392513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.525277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.550701] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2539.673383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2539.697619] LNet: 136844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.708585] LNetError: 136844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.733941] LNet: Removed LNI 192.168.204.40@tcp [ 2539.763623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.823414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2539.852943] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2545.765086] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:55:38 (1787788538) [ 2553.055321] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:55:46 (1787788546) [ 2559.538237] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:55:52 (1787788552) [ 2567.334630] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:55:59 (1787788559) [ 2575.132922] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:56:07 (1787788567) [ 2584.035486] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:56:16 (1787788576) [ 2592.104988] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:56:24 (1787788584) [ 2600.262825] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:56:32 (1787788592) [ 2607.644704] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:56:40 (1787788600) [ 2615.458885] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:56:48 (1787788608) [ 2623.360043] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:56:55 (1787788615) [ 2631.316946] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:57:03 (1787788623) [ 2639.364478] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2410 sec ========== 19:57:12 (1787788632) [ 2639.865303] LNet: 144309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2639.875761] LNetError: 144309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2639.896869] LNet: Removed LNI 192.168.204.40@tcp [ 2640.694825] Key type .llcrypt unregistered [ 2640.704768] Key type ._llcrypt unregistered [ 2653.974503] Key type ._llcrypt registered [ 2654.082443] Key type .llcrypt registered [ 2654.409429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2654.423334] alg: No test for adler32 (adler32-zlib) [ 2655.474900] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 2655.791451] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2657.527213] Key type lgssc registered [ 2658.894974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2723.159664] Lustre: Mounted lustre-client - version 2.17.57_53_g458ba6d [ 2727.716943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2736.075734] Lustre: Unmounted lustre-client [ 2767.651874] Key type lgssc unregistered [ 2767.979582] LNet: 147276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2767.990166] LNetError: 147276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2768.015514] LNet: Removed LNI 192.168.204.40@tcp [ 2768.866516] Key type .llcrypt unregistered [ 2768.868759] Key type ._llcrypt unregistered