[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 426236929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003034] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008357] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012104] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014032] SELinux: Initializing. [ 0.014765] *** VALIDATE selinux *** [ 0.022681] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026876] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028186] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.030551] *** VALIDATE proc *** [ 0.031292] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.034211] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035270] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041098] debug: unmapping init [mem 0xffffffffb1659000-0xffffffffb1660fff] [ 0.043936] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044662] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049012] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052314] rcu: Hierarchical SRCU implementation. [ 0.054325] smp: Bringing up secondary CPUs ... [ 0.055605] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.060088] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147160] node 0 deferred pages initialised in 80ms [ 0.150324] devtmpfs: initialized [ 0.151216] x86/mm: Memory block size: 128MB [ 0.153935] gcov: version magic: 0x41383552 [ 0.155132] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157243] pinctrl core: initialized pinctrl subsystem [ 0.158319] [ 0.158944] ************************************************************* [ 0.159014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160014] ** ** [ 0.161013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162016] ** ** [ 0.163018] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165014] ** your system. ** [ 0.166012] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.168017] ** kernel, report this immediately to your vendor! ** [ 0.169019] ** ** [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ************************************************************* [ 0.172815] NET: Registered protocol family 16 [ 0.173463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176445] cpuidle: using governor menu [ 0.179304] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183987] PCI: Using configuration type 1 for base access [ 0.188122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204364] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211105] cryptd: max_cpu_qlen set to 1000 [ 0.214229] ACPI: Added _OSI(Module Device) [ 0.216030] ACPI: Added _OSI(Processor Device) [ 0.219019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226205] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232600] ACPI: Interpreter enabled [ 0.233719] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235015] ACPI: Using IOAPIC for interrupt routing [ 0.238166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243376] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255874] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270385] acpiphp: Slot [2] registered [ 0.272121] acpiphp: Slot [5] registered [ 0.274118] acpiphp: Slot [6] registered [ 0.275121] acpiphp: Slot [3] registered [ 0.277097] acpiphp: Slot [4] registered [ 0.278099] acpiphp: Slot [7] registered [ 0.280098] acpiphp: Slot [8] registered [ 0.282100] acpiphp: Slot [9] registered [ 0.284315] acpiphp: Slot [10] registered [ 0.287322] acpiphp: Slot [11] registered [ 0.290171] acpiphp: Slot [12] registered [ 0.293125] acpiphp: Slot [13] registered [ 0.295235] acpiphp: Slot [14] registered [ 0.297158] acpiphp: Slot [15] registered [ 0.299127] acpiphp: Slot [16] registered [ 0.302127] acpiphp: Slot [17] registered [ 0.303106] acpiphp: Slot [18] registered [ 0.305127] acpiphp: Slot [19] registered [ 0.307281] acpiphp: Slot [20] registered [ 0.309151] acpiphp: Slot [21] registered [ 0.312190] acpiphp: Slot [22] registered [ 0.314107] acpiphp: Slot [23] registered [ 0.316127] acpiphp: Slot [24] registered [ 0.318120] acpiphp: Slot [25] registered [ 0.320125] acpiphp: Slot [26] registered [ 0.322268] acpiphp: Slot [27] registered [ 0.324114] acpiphp: Slot [28] registered [ 0.326119] acpiphp: Slot [29] registered [ 0.328117] acpiphp: Slot [30] registered [ 0.330184] acpiphp: Slot [31] registered [ 0.332084] PCI host bridge to bus 0000:00 [ 0.334023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357081] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360452] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.375323] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389346] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.392777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.414879] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.419015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424581] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473256] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.521045] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.523643] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.527440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.529364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.531220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.537092] iommu: Default domain type: Passthrough [ 0.539667] SCSI subsystem initialized [ 0.541133] ACPI: bus type USB registered [ 0.543155] usbcore: registered new interface driver usbfs [ 0.545098] usbcore: registered new interface driver hub [ 0.547106] usbcore: registered new device driver usb [ 0.549142] pps_core: LinuxPPS API ver. 1 registered [ 0.552012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.556106] PTP clock support registered [ 0.559042] EDAC MC: Ver: 3.0.0 [ 0.561163] PCI: Using ACPI for IRQ routing [ 0.563237] NetLabel: Initializing [ 0.564027] NetLabel: domain hash size = 128 [ 0.566013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.568091] NetLabel: unlabeled traffic allowed by default [ 0.571364] vgaarb: loaded [ 0.573184] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.575013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.581703] clocksource: Switched to clocksource kvm-clock [ 0.691302] VFS: Disk quotas dquot_6.6.0 [ 0.692952] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.696313] *** VALIDATE ramfs *** [ 0.697666] *** VALIDATE hugetlbfs *** [ 0.699328] pnp: PnP ACPI init [ 0.702482] pnp: PnP ACPI: found 6 devices [ 0.721978] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.725305] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.727120] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729494] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.731320] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.733386] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.736355] NET: Registered protocol family 2 [ 0.738949] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.743891] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.746592] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.752431] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755666] TCP: Hash tables configured (established 65536 bind 65536) [ 0.758676] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.761916] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764912] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.767947] NET: Registered protocol family 1 [ 0.770536] RPC: Registered named UNIX socket transport module. [ 0.772696] RPC: Registered udp transport module. [ 0.774270] RPC: Registered tcp transport module. [ 0.776085] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.778529] NET: Registered protocol family 44 [ 0.780286] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.782576] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.784804] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.787171] PCI: CLS 0 bytes, default 64 [ 0.788920] Unpacking initramfs... [ 2.199619] debug: unmapping init [mem 0xffff9268bcc64000-0xffff9268bffcffff] [ 2.203792] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.205409] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.208131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.479351] Initialise system trusted keyrings [ 3.485797] Key type blacklist registered [ 3.495397] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.520826] zbud: loaded [ 3.533638] *** VALIDATE nfs *** [ 3.534985] *** VALIDATE nfs4 *** [ 3.542443] pstore: using deflate compression [ 3.553435] Platform Keyring initialized [ 3.924653] NET: Registered protocol family 38 [ 3.927192] Key type asymmetric registered [ 3.931817] Asymmetric key parser 'x509' registered [ 3.943403] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.966986] io scheduler mq-deadline registered [ 3.971350] io scheduler kyber registered [ 3.977772] io scheduler bfq registered [ 3.983792] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.991274] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.001195] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.009743] ACPI: Power Button [PWRF] [ 4.029402] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.055035] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.077557] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.123230] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.191878] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.223994] Non-volatile memory driver v1.3 [ 4.228895] Linux agpgart interface v0.103 [ 4.366322] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.373563] vda: detected capacity change from 0 to 74698752 [ 4.418460] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.432019] vdb: detected capacity change from 0 to 1073741824 [ 4.451333] libphy: Fixed MDIO Bus: probed [ 4.472711] usbcore: registered new interface driver usbserial_generic [ 4.476470] usbserial: USB Serial support registered for generic [ 4.479635] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.485294] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.487824] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.503351] mousedev: PS/2 mouse device common for all mice [ 4.516338] rtc_cmos 00:05: RTC can wake from S4 [ 4.524557] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.543430] rtc_cmos 00:05: registered as rtc0 [ 4.548410] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.553301] intel_pstate: CPU model not supported [ 4.559659] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.567175] hid: raw HID events driver (C) Jiri Kosina [ 4.571297] usbcore: registered new interface driver usbhid [ 4.572891] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.582476] usbhid: USB HID core driver [ 4.593798] drop_monitor: Initializing network drop monitor service [ 4.598450] Initializing XFRM netlink socket [ 4.604299] NET: Registered protocol family 10 [ 4.613688] Segment Routing with IPv6 [ 4.618148] NET: Registered protocol family 17 [ 4.623900] mpls_gso: MPLS GSO support [ 4.641606] RAS: Correctable Errors collector initialized. [ 4.647968] AVX version of gcm_enc/dec engaged. [ 4.651046] AES CTR mode by8 optimization enabled [ 4.866704] sched_clock: Marking stable (4866667952, 0)->(5803898918, -937230966) [ 4.875736] registered taskstats version 1 [ 4.878664] Loading compiled-in X.509 certificates [ 4.882102] zswap: loaded using pool lzo/zbud [ 4.976328] Key type big_key registered [ 5.013283] Key type encrypted registered [ 5.016367] ima: No TPM chip found, activating TPM-bypass! [ 5.028157] ima: Allocated hash algorithm: sha1 [ 5.030034] ima: No architecture policies found [ 5.032359] evm: Initialising EVM extended attributes: [ 5.034314] evm: security.selinux [ 5.039841] evm: security.ima [ 5.041314] evm: security.capability [ 5.042719] evm: HMAC attrs: 0x1 [ 5.060252] rtc_cmos 00:05: setting system clock to 2026-08-14 21:42:29 UTC (1786743749) [ 5.077353] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 5.081911] debug: unmapping init [mem 0xffffffffb1382000-0xffffffffb1658fff] [ 5.095183] Write protecting the kernel read-only data: 28672k [ 5.098839] debug: unmapping init [mem 0xffffffffafa03000-0xffffffffafbfffff] [ 5.102054] debug: unmapping init [mem 0xffffffffb0314000-0xffffffffb03fffff] [ 5.141930] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.151123] systemd[1]: Detected virtualization kvm. [ 5.152973] systemd[1]: Detected architecture x86-64. [ 5.155584] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.189359] systemd[1]: No hostname configured. [ 5.193684] systemd[1]: Set hostname to . [ 5.200708] random: systemd: uninitialized urandom read (16 bytes read) [ 5.205399] systemd[1]: Initializing machine ID from random generator. [ 5.399184] random: ln: uninitialized urandom read (6 bytes read) [ 5.704841] random: systemd: uninitialized urandom read (16 bytes read) [ 5.719944] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.736883] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.769834] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.583394] device-mapper: uevent: version 1.0.3 [ 7.586685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.790360] virtio_net virtio0 ens2: renamed from eth0 [ 10.250654] scsi host0: ata_piix [ 10.290596] scsi host1: ata_piix [ 10.295547] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.300424] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.569975] random: crng init done [ 15.571908] random: 7 urandom warning(s) missed due to ratelimiting [ 18.071373] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.484418] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.377334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.273989] SELinux: Disabled at runtime. [ 24.393745] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.404390] systemd[1]: Detected virtualization kvm. [ 24.406812] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.229146] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.248334] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.267296] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.287813] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.309751] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.371489] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.407872] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 26.511363] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.245482] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.168339] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.204527] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.458713] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.774548] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.385646] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 36.653438] NFS: Registering the id_resolver key type [ 36.656398] Key type id_resolver registered [ 36.658359] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-client login: [ 59.023069] hrtimer: interrupt took 7542822 ns [ 108.910815] libcfs: loading out-of-tree module taints kernel. [ 109.080816] Key type ._llcrypt registered [ 109.087154] Key type .llcrypt registered [ 109.753369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 109.770779] alg: No test for adler32 (adler32-zlib) [ 111.188287] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 112.206520] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 113.927158] Key type lgssc registered [ 115.509787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 255.386204] Lustre: Mounted lustre-client - version 2.17.57_1_g59aa375 [ 260.284412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.113500] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing check_logdir /tmp/testlogs/ [ 277.278494] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing yml_node [ 281.055183] Lustre: lustre-OST0000-osc-ffff92691826e000: disconnect after 23s idle [ 281.778059] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 285.020510] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 287.601679] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 289.130940] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 17:47:12 EDT 2026 [ 305.394739] Lustre: Unmounted lustre-client [ 343.757915] Key type lgssc unregistered [ 344.118927] LNet: 8417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.135107] LNetError: 8417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.191269] LNet: Removed LNI 192.168.203.23@tcp [ 346.722856] Key type .llcrypt unregistered [ 346.731341] Key type ._llcrypt unregistered [ 380.987441] Key type ._llcrypt registered [ 380.991672] Key type .llcrypt registered [ 381.301880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 381.332555] alg: No test for adler32 (adler32-zlib) [ 382.442508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 382.540962] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 382.583764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 387.442812] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 387.589232] LNet: 9299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.606639] LNetError: 9299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.634740] LNet: Removed LNI 192.168.203.23@tcp [ 388.516708] Key type .llcrypt unregistered [ 388.518480] Key type ._llcrypt unregistered [ 396.172449] Key type ._llcrypt registered [ 396.175515] Key type .llcrypt registered [ 397.320928] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:49:00 (1786744140) [ 397.841493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.867637] alg: No test for adler32 (adler32-zlib) [ 398.912674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.977706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.000959] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.041580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.162636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 399.198266] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 407.717659] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:49:10 (1786744150) [ 407.868951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.896672] LNet: 10310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.924446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.021727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 416.073737] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:49:18 (1786744158) [ 416.227463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.275672] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.319240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.407559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 424.133454] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:49:27 (1786744167) [ 424.473368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.518899] LNet: 11480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.559685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.700809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 432.535177] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:49:35 (1786744175) [ 432.762690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.800224] LNet: 12065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.837134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.910704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 433.268869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 440.778665] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:49:43 (1786744183) [ 441.026937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.118759] LNet: 12652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.175414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.322277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 442.226403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 443.509486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 451.591485] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:49:54 (1786744194) [ 451.824913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.860293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.989023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 459.874721] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:50:02 (1786744202) [ 460.093555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.122016] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.131488] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 460.162688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.205947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 460.250327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 460.789198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 460.832835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 461.355760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 461.404296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 461.904388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 461.929243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 462.573587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 462.665369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.212100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 463.250521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 463.590088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 463.631877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.135830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 464.206360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 464.903485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 464.964711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.246241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 465.268623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 465.553652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 465.577902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 465.965719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 466.020689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.634709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 466.692140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 474.029365] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:50:17 (1786744217) [ 474.216347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.283583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.386957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 474.416027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 481.688456] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:50:24 (1786744224) [ 481.999810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.043634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.130702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 482.167755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 490.324481] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:50:32 (1786744232) [ 490.407574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.472576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.576062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 490.612697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 497.222886] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:50:40 (1786744240) [ 497.299832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.321496] LNet: 16272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.329908] LNet: 16272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 497.385288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.476553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 497.552777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 506.467825] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:50:49 (1786744249) [ 506.565975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.666964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.812751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 506.855290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 513.172384] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:50:56 (1786744256) [ 513.279946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.343901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.449951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 513.509732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 521.218554] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:51:04 (1786744264) [ 521.474249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.537529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.671516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 522.254594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 530.533278] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:51:13 (1786744273) [ 530.701516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.769529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.172013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 531.385868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 537.812472] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:51:20 (1786744280) [ 537.911357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.980242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.165313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 538.512577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 544.497328] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:51:27 (1786744287) [ 544.772443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.823416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.281020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.533882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.948879] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:51:35 (1786744295) [ 552.157373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.213964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.347904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.415639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 559.595481] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:51:42 (1786744302) [ 559.981376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.012398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.073158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.112944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 567.482366] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:51:50 (1786744310) [ 567.689711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.733101] LNet: 21716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.751720] LNet: 21716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 567.814961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.928657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 575.405887] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:51:58 (1786744318) [ 575.559270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.612150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.664428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 583.090734] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:52:06 (1786744326) [ 583.450367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.523387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.655265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 584.422101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 591.046167] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:52:14 (1786744334) [ 591.149697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.219652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.349498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 591.394913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 599.293950] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:52:22 (1786744342) [ 599.346420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.379706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.480778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 599.522785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 606.328958] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:52:29 (1786744349) [ 606.458214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.509433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.611920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 606.640587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 614.498340] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:52:37 (1786744357) [ 614.623645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.688695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.769428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 614.806136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 621.933118] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:52:45 (1786744365) [ 622.138648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.170267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.248383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 622.339901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 622.355826] LNetError: 25834:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 622.393492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 622.457961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 628.190346] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:52:51 (1786744371) [ 628.571146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.633760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.767453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dsc4rGs4Rw [ 628.907989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dsc4rGs4Rw [ 628.968371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Dsc4rGs4Rw [ 634.558412] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:52:57 (1786744377) [ 634.663562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.719444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.780556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 634.807767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 634.858167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 641.002763] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:53:04 (1786744384) [ 641.095157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.174589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.284612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.310126] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 642.658590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 643.892750] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 645.871176] Key type lgssc registered [ 647.230214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.939504] LNetError: 27594:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.23@tcp99 has no available nets [ 653.089963] Key type lgssc unregistered [ 653.410861] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.428127] LNet: Removed LNI 192.168.203.23@tcp [ 654.020095] Key type .llcrypt unregistered [ 654.024673] Key type ._llcrypt unregistered [ 660.480718] Key type ._llcrypt registered [ 660.485697] Key type .llcrypt registered [ 661.734645] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:53:24 (1786744404) [ 662.247469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.282958] alg: No test for adler32 (adler32-zlib) [ 663.216209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.309115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 666.479275] LNet: Message forwarding will be enabled in 2s [ 669.535128] LNet: Message forwarding enabled [ 669.954315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 669.978654] LNet: Message forwarding will stop in 2s to 6s [ 673.328222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 674.033295] LNet: Message forwarding disabled [ 675.584292] LNet: Message forwarding will be enabled in 2s [ 679.135847] LNet: Message forwarding enabled [ 682.005906] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:53:45 (1786744425) [ 682.092638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.109707] LNet: 30096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.425638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.489881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 689.466219] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:53:52 (1786744432) [ 689.552397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.578384] LNet: 30678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.606687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.704692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 689.751113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 689.780477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 689.804721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 689.835815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 689.858355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 689.884956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 689.917247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 689.944164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 689.979408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 690.004590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 690.028880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 690.052349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 690.080273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 690.107256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 690.137472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 690.166750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 690.199302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 690.228331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 690.251371] LNetError: 30727:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 696.448487] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:53:58 (1786744438) [ 696.648084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.694302] LNet: 31295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.754703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.880652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.823202] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:54:06 (1786744446) [ 705.984674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg323-server [ 707.837682] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:54:10 (1786744450) [ 708.043842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.109952] LNet: 32227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.196539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.269343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.301441] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 709.163954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[24-28]@tcp [ 709.178338] LNet: 32363:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 709.224852] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.24@tcp is being used as a gateway but routing feature is not turned on [ 709.240733] LNetError: 32234:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.24@tcp has gone from up to down [ 710.248932] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.24@tcp is being used as a gateway but routing feature is not turned on [ 710.262976] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 711.284407] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 711.303660] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 713.405632] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.28@tcp is being used as a gateway but routing feature is not turned on [ 713.429667] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 716.795356] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:54:19 (1786744459) [ 717.476584] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.25@tcp is being used as a gateway but routing feature is not turned on [ 717.502905] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 718.678869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg323-server [ 720.895525] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:54:23 (1786744463) [ 721.003227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.033799] LNet: 33291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.046252] LNetError: 33291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.086201] LNet: Removed LNI 192.168.203.23@tcp [ 721.125687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.199053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.221663] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 721.891991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[24-29/2]@tcp [ 721.922768] LNet: 33416:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 721.937519] LNet: 33416:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 721.969569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[24-29/2]@tcp [ 721.987230] LNetError: 33303:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.24@tcp has gone from up to down [ 722.008426] LNetError: 33303:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 728.517634] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:54:31 (1786744471) [ 728.640789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.682684] LNetError: 34005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.702968] LNet: Removed LNI 192.168.203.23@tcp [ 728.762316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.836124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 728.902860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.969445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.032325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.084280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.103567] LNet: 34035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.112647] LNet: 34035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 729.144794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.216953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 729.312524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.343326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.421600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.498210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.518258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.562732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 729.647825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.691672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.756297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.875173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.916680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.992283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 730.092768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.143879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.212460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.298336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.329551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.384890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 730.450590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.473960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.536690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.622179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.655400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.707500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 730.765209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.811522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.882423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.694678] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:54:39 (1786744479) [ 738.227659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg323-server [ 739.972252] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:54:43 (1786744483) [ 741.476410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg323-server [ 742.853473] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:54:46 (1786744486) [ 743.673189] Key type .llcrypt unregistered [ 743.677542] Key type ._llcrypt unregistered [ 749.190931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.320266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.323812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.328864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.892131] Key type ._llcrypt registered [ 751.900207] Key type .llcrypt registered [ 752.131027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.152670] alg: No test for adler32 (adler32-zlib) [ 753.018542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.089806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.127743] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 753.772281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 753.802548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 755.693871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 755.722810] LNet: Removed LNI 10.1.2.3@tcp [ 755.867611] LNet: 36138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.874549] LNetError: 36138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.888182] LNet: Removed LNI 192.168.203.23@tcp [ 756.644493] Key type .llcrypt unregistered [ 756.648132] Key type ._llcrypt unregistered [ 767.729859] Key type ._llcrypt registered [ 767.736513] Key type .llcrypt registered [ 769.096104] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:55:11 (1786744511) [ 769.560283] Key type .llcrypt unregistered [ 769.563204] Key type ._llcrypt unregistered [ 776.039141] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.135037] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.151773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.165990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.861713] Key type ._llcrypt registered [ 778.864249] Key type .llcrypt registered [ 779.096790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.128274] alg: No test for adler32 (adler32-zlib) [ 780.041400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.100854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.128706] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 781.020516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.044741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.177901] LNet: 37642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.190596] LNetError: 37642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 783.197640] LNet: Removed LNI 10.1.2.3@tcp [ 783.206440] LNet: Removed LNI 192.168.203.23@tcp [ 783.787621] Key type .llcrypt unregistered [ 783.789673] Key type ._llcrypt unregistered [ 793.683629] Key type ._llcrypt registered [ 793.687143] Key type .llcrypt registered [ 794.737537] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:55:37 (1786744537) [ 795.468670] Key type .llcrypt unregistered [ 795.470321] Key type ._llcrypt unregistered [ 800.763626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.906665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.916348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.924710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.522718] Key type ._llcrypt registered [ 803.524893] Key type .llcrypt registered [ 803.674945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.688604] alg: No test for adler32 (adler32-zlib) [ 804.615237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.684949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 804.703084] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 806.224210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.237986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.912781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 807.933568] LNet: Removed LNI 10.1.2.3@tcp [ 807.951702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 808.007699] LNet: Removed LNI 10.1.2.31@tcp [ 808.043746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 808.075398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.728994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 808.742839] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 809.437947] LNet: 39569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.442905] LNetError: 39569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 809.448359] LNet: Removed LNI 10.1.2.31@tcp [ 809.454796] LNet: Removed LNI 10.1.2.3@tcp [ 809.932861] Key type .llcrypt unregistered [ 809.934538] Key type ._llcrypt unregistered [ 820.426200] Key type ._llcrypt registered [ 820.428812] Key type .llcrypt registered [ 821.676882] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:56:04 (1786744564) [ 821.994539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.015268] alg: No test for adler32 (adler32-zlib) [ 822.929626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.982162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.999174] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 823.635803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.654335] LNet: 40525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.659313] LNetError: 40525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.673762] LNet: Removed LNI 192.168.203.23@tcp [ 823.693483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.732510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.23@tcp [ 823.760217] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 824.454193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.23@tcp [ 824.482435] LNet: Removed LNI 192.168.203.23@tcp [ 824.527801] LNet: 40660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.102698] Key type .llcrypt unregistered [ 825.105295] Key type ._llcrypt unregistered [ 834.656203] Key type ._llcrypt registered [ 834.657723] Key type .llcrypt registered [ 835.962303] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:56:18 (1786744578) [ 837.572356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg323-server [ 839.062702] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:56:22 (1786744582) [ 839.543221] Key type .llcrypt unregistered [ 839.544809] Key type ._llcrypt unregistered [ 844.307704] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.452158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.455761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.460030] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.866435] Key type ._llcrypt registered [ 846.869200] Key type .llcrypt registered [ 847.004785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.015439] alg: No test for adler32 (adler32-zlib) [ 847.905367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.946611] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 848.466805] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 849.792188] LNet: 42501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.806401] LNetError: 42501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.810770] LNet: Removed LNI 192.168.203.23@tcp1 [ 849.830943] LNet: Removed LNI 10.1.2.3@tcp2 [ 850.401479] Key type .llcrypt unregistered [ 850.407912] Key type ._llcrypt unregistered [ 860.304862] Key type ._llcrypt registered [ 860.308340] Key type .llcrypt registered [ 861.446261] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:56:44 (1786744604) [ 861.945881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.959247] alg: No test for adler32 (adler32-zlib) [ 862.903524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.957558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.998918] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 863.703756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.719284] LNet: 43472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.722153] LNetError: 43472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.740522] LNet: Removed LNI 192.168.203.23@tcp [ 863.756462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.797891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 863.814711] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 870.078786] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:56:53 (1786744613) [ 870.347909] LNet: 44064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.361754] LNetError: 44064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.382365] LNet: Removed LNI 192.168.203.23@tcp [ 870.921684] Key type .llcrypt unregistered [ 870.923789] Key type ._llcrypt unregistered [ 876.659187] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.753355] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.765235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.772313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.337469] Key type ._llcrypt registered [ 879.339856] Key type .llcrypt registered [ 879.483474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.503459] alg: No test for adler32 (adler32-zlib) [ 880.420516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.468412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.501649] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 881.323670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 881.347035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.513420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.528369] LNet: 44810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.533378] LNetError: 44810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.538888] LNet: Removed LNI 10.1.2.3@tcp [ 883.547909] LNet: Removed LNI 192.168.203.23@tcp [ 883.572380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.654617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.678870] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 883.683397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.297742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.319485] LNet: 44837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.325745] LNetError: 44837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.334824] LNetError: 44837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.347449] LNet: Removed LNI 10.1.2.3@tcp [ 884.359644] LNet: Removed LNI 192.168.203.23@tcp [ 884.406737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.518730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.552344] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 884.558755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.074285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 885.090827] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 885.936524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.953803] LNet: 44999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.967550] LNetError: 44999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.976176] LNetError: 44999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.982434] LNet: Removed LNI 10.1.2.3@tcp [ 885.984802] LNet: Removed LNI 192.168.203.23@tcp [ 885.997552] LNet: Removed LNI 192.168.203.23@tcp2 [ 886.013903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.065165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 886.106485] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 886.113902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.120227] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 891.850822] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:57:15 (1786744635) [ 891.928832] LNet: 45587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.937367] LNetError: 45587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.942030] LNetError: 45587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 891.948773] LNet: Removed LNI 10.1.2.3@tcp [ 891.955906] LNet: Removed LNI 192.168.203.23@tcp [ 891.975488] LNet: Removed LNI 192.168.203.23@tcp2 [ 892.627846] Key type .llcrypt unregistered [ 892.630311] Key type ._llcrypt unregistered [ 897.790940] Key type ._llcrypt registered [ 897.793519] Key type .llcrypt registered [ 897.971218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.985785] alg: No test for adler32 (adler32-zlib) [ 898.918758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.013637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.058518] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 899.773533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.803391] LNet: 45994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.811357] LNetError: 45994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.835302] LNet: Removed LNI 192.168.203.23@tcp [ 899.852695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.936838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 899.971600] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 900.491805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.526456] LNet: 46024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.545778] LNetError: 46024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.573866] LNet: Removed LNI 192.168.203.23@tcp [ 900.612162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.685546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 900.715900] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 901.343697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.373828] LNet: Removed LNI 192.168.203.23@tcp [ 901.395848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.476357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 901.532301] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 902.032529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 902.203936] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 902.212370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 902.228206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.731529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.779488] LNet: 46136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.792130] LNet: 46136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 904.808983] LNetError: 46136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.818831] LNetError: 46136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 904.840271] LNet: Removed LNI 192.168.203.23@tcp [ 904.850662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.893837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 904.910567] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 910.894229] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:57:34 (1786744654) [ 910.981204] LNet: 46725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.995607] LNetError: 46725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.015928] LNet: Removed LNI 192.168.203.23@tcp [ 911.559651] Key type .llcrypt unregistered [ 911.562797] Key type ._llcrypt unregistered [ 917.187616] Key type ._llcrypt registered [ 917.191858] Key type .llcrypt registered [ 917.333995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.351834] alg: No test for adler32 (adler32-zlib) [ 918.289985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.347123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.383982] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 919.191872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.209519] LNet: 47135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.216958] LNetError: 47135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.238854] LNet: Removed LNI 192.168.203.23@tcp [ 919.263523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.310521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 919.337354] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 925.523929] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:57:48 (1786744668) [ 925.610450] LNet: 47718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.618223] LNetError: 47718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.636409] LNet: Removed LNI 192.168.203.23@tcp [ 926.202684] Key type .llcrypt unregistered [ 926.204644] Key type ._llcrypt unregistered [ 931.779728] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.924259] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.931022] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.943120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.520746] Key type ._llcrypt registered [ 934.523648] Key type .llcrypt registered [ 934.693234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.707809] alg: No test for adler32 (adler32-zlib) [ 935.634410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.675975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.694319] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 936.351308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.368706] LNet: 48182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.372990] LNetError: 48182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.389332] LNet: Removed LNI 192.168.203.23@tcp [ 936.403652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.471633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 936.511759] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 941.950499] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:58:05 (1786744685) [ 942.146772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.172680] LNet: 48780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.182675] LNetError: 48780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.201983] LNet: Removed LNI 192.168.203.23@tcp [ 942.232474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.360123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.082899] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:58:11 (1786744691) [ 948.262959] LNet: 49362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.920565] Key type .llcrypt unregistered [ 948.922117] Key type ._llcrypt unregistered [ 954.432462] Key type ._llcrypt registered [ 954.435189] Key type .llcrypt registered [ 954.638679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.652550] alg: No test for adler32 (adler32-zlib) [ 955.601088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.681219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.710604] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 956.360823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.377234] LNet: 49764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.382748] LNetError: 49764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.399670] LNet: Removed LNI 192.168.203.23@tcp [ 956.417672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.474804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 956.502198] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 962.500446] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:58:25 (1786744705) [ 962.601252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.619566] LNet: 50350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.623740] LNetError: 50350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.640077] LNet: Removed LNI 192.168.203.23@tcp [ 962.668024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.745678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 969.209165] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:58:32 (1786744712) [ 970.890940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 972.405274] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:58:35 (1786744715) [ 973.845679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 975.440843] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:58:38 (1786744718) [ 976.858925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 978.566391] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:58:41 (1786744721) [ 979.748799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 981.358542] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:58:44 (1786744724) [ 983.469190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 985.155949] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 17:58:48 (1786744728) [ 985.724651] LNet: 52408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.551672] Key type .llcrypt unregistered [ 986.554489] Key type ._llcrypt unregistered [ 993.354977] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.525767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.538842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.548027] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.046784] Key type ._llcrypt registered [ 996.051937] Key type .llcrypt registered [ 996.259078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.269938] alg: No test for adler32 (adler32-zlib) [ 997.215230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.304143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.326980] LNet: Added LNI 192.168.203.23@tcp [8/512/0/180] [ 997.332187] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 997.337914] LNet: 52735:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 997.465379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.505909] LNet: 52754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.510896] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 997.520832] LNet: Removed LNI 10.1.2.3@tcp [ 997.522883] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.535121] LNet: Removed LNI 192.168.203.23@tcp [ 997.568507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.629274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.671448] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 997.745796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.770123] LNet: 52778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.775709] LNetError: 52778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.798882] LNet: Removed LNI 192.168.203.23@tcp [ 997.825712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.905546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.934576] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 997.937521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 997.942660] LNet: 52786:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.049188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.064862] LNet: 52804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.072376] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.078059] LNet: Removed LNI 10.1.2.3@tcp [ 998.080505] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.092516] LNet: Removed LNI 192.168.203.23@tcp [ 998.103790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.165655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.186559] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 998.191700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.194994] LNet: 52812:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.349309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.366450] LNet: 52838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.373394] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.378631] LNet: Removed LNI 10.1.2.3@tcp [ 998.381239] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.392455] LNet: Removed LNI 192.168.203.23@tcp [ 998.406405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.455938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.473633] LNet: Added LNI 192.168.203.23@tcp [8/256/0/182] [ 998.479041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 998.563922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.578029] LNet: 52866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.583735] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.590330] LNet: Removed LNI 10.1.2.3@tcp [ 998.593248] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.607974] LNet: Removed LNI 192.168.203.23@tcp [ 998.625186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.681408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.699606] LNet: Added LNI 192.168.203.23@tcp [8/256/0/182] [ 998.704336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 998.707424] LNet: 52874:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.769258] LNet: Removed LNI 10.1.2.3@tcp [ 998.778557] LNet: Removed LNI 192.168.203.23@tcp [ 999.254093] Key type .llcrypt unregistered [ 999.256415] Key type ._llcrypt unregistered [ 1010.531458] Key type ._llcrypt registered [ 1010.535820] Key type .llcrypt registered [ 1011.723352] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 17:59:14 (1786744754) [ 1013.087225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1015.234243] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:59:17 (1786744757) [ 1015.597746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.611888] alg: No test for adler32 (adler32-zlib) [ 1016.542148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.576836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1016.591253] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1017.324134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.23 [ 1017.353790] LNet: 54144:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1017.364390] LNet: Removed LNI 192.168.203.23@tcp [ 1023.105920] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:59:26 (1786744766) [ 1023.369946] LNet: 54714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.375904] LNetError: 54714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.401935] LNet: Removed LNI 192.168.203.23@tcp [ 1023.954738] Key type .llcrypt unregistered [ 1023.957160] Key type ._llcrypt unregistered [ 1029.881645] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1030.069585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.711457] Key type ._llcrypt registered [ 1032.714497] Key type .llcrypt registered [ 1032.888188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.901245] alg: No test for adler32 (adler32-zlib) [ 1034.201726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1039.160576] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:59:42 (1786744782) [ 1039.267126] LNet: 55625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.274358] LNetError: 55625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1039.288164] LNet: Removed LNI 10.1.2.3@tcp [ 1039.848131] Key type .llcrypt unregistered [ 1039.849757] Key type ._llcrypt unregistered [ 1046.066624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1046.315143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1046.318218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1046.327367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.179634] Key type ._llcrypt registered [ 1049.181910] Key type .llcrypt registered [ 1049.346606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.364475] alg: No test for adler32 (adler32-zlib) [ 1050.335796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.849244] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:59:59 (1786744799) [ 1058.462750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1060.465860] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:00:03 (1786744803) [ 1060.583304] LNet: 56813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.602308] LNetError: 56813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.632799] LNet: Removed LNI 10.1.2.3@tcp [ 1061.482443] Key type .llcrypt unregistered [ 1061.487210] Key type ._llcrypt unregistered [ 1068.647363] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1068.992847] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1069.000138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1069.008855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1071.925936] Key type ._llcrypt registered [ 1071.930831] Key type .llcrypt registered [ 1072.198718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.219946] alg: No test for adler32 (adler32-zlib) [ 1073.182173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.244322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1082.329070] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:00:25 (1786744825) [ 1082.939154] LNet: 58056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.946385] LNetError: 58056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.962572] LNet: Removed LNI 10.1.2.3@tcp [ 1083.787302] Key type .llcrypt unregistered [ 1083.790938] Key type ._llcrypt unregistered [ 1090.751381] Key type ._llcrypt registered [ 1090.754434] Key type .llcrypt registered [ 1090.992408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.013958] alg: No test for adler32 (adler32-zlib) [ 1091.935592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.961885] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1099.287894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1099.864849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_interrupt -r 1 [ 1099.921160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_interrupt -r 1 [ 1099.978274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1100.016979] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1100.191594] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1100.254903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.314889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.352263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.378851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.405359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.428995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.454256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.482154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.496067] LNet: 58498:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.516898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.575829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.622530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.678454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.723496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.774970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.810970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.312362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_dropped -r 1 [ 1101.357429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_dropped -r 1 [ 1101.388871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1101.409265] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1101.652625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.682645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.728656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.763861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.804417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.857687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.883554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.911950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.935258] LNet: 58607:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.952369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.985561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.024090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.066801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.101603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.139732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.186574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.730972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_aborted -r 1 [ 1102.766832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_aborted -r 1 [ 1102.793564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1102.811260] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1103.022646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.053551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.092227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.126223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.182054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.256181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.351572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.425165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.446061] LNet: 58717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.476489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.537586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.587946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.629285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.659780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.694745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.736497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.284294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_no_route -r 1 [ 1104.336342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_no_route -r 1 [ 1104.366214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1104.603322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.668174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.709942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.744162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.777393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.816646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.870851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.925776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.983819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.027951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.055364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.084546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.111805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.142510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.173824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.805633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_timeout -r 1 [ 1105.876670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_timeout -r 1 [ 1105.932895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1105.979476] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1105.985916] LNet: Skipped 2 previous similar messages [ 1106.183870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.219812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.241968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.265669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.291431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.344467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.394188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.429177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.444697] LNet: 58935:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.453388] LNet: 58935:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1106.473983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.515478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.547058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.602973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.650871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.689282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.733557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.174694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_error -r 1 [ 1107.215800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_error -r 1 [ 1107.250617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1107.542940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.599609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.659576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.698768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.742724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.781613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.833874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.869887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.914962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.962949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.005217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.064514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.120472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.160795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.202759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.565838] LNet: 59090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.580765] LNetError: 59090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.606817] LNet: Removed LNI 192.168.203.23@tcp [ 1110.049513] Key type .llcrypt unregistered [ 1110.050941] Key type ._llcrypt unregistered [ 1127.643290] Key type ._llcrypt registered [ 1127.649578] Key type .llcrypt registered [ 1128.874765] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:01:11 (1786744871) [ 1129.712933] Key type .llcrypt unregistered [ 1129.728844] Key type ._llcrypt unregistered [ 1135.654746] Key type ._llcrypt registered [ 1135.657186] Key type .llcrypt registered [ 1135.872922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.887983] alg: No test for adler32 (adler32-zlib) [ 1136.813775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.851080] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1145.032768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1155.205506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.260404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1155.280079] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1156.466330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_interrupt -r 1 [ 1156.506151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_interrupt -r 1 [ 1156.568477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_interrupt -r 1 [ 1156.626397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_interrupt -r 1 [ 1156.681598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_interrupt -r 1 [ 1156.729239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_interrupt -r 1 [ 1156.771336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_interrupt -r 1 [ 1156.830452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_interrupt -r 1 [ 1156.883349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1156.919975] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1156.928170] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1157.180635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.230154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.261859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.286916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.309736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.331878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.359347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.384605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.401677] LNet: 60577:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.417064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.439125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.466491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.487285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.508397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.530580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.553241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.046598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_dropped -r 1 [ 1158.078795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_dropped -r 1 [ 1158.103628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_dropped -r 1 [ 1158.134243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_dropped -r 1 [ 1158.167817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_dropped -r 1 [ 1158.211467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_dropped -r 1 [ 1158.261780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_dropped -r 1 [ 1158.299968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_dropped -r 1 [ 1158.333737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1158.362619] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1158.367602] LNet: Skipped 2 previous similar messages [ 1158.624938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.695470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.751157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.798432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.831960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.854210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.887987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.909732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.928171] LNet: 60698:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.947329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.984471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.018433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.053767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.101758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.133976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.160386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.575477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_aborted -r 1 [ 1159.599496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_aborted -r 1 [ 1159.627159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_aborted -r 1 [ 1159.653785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_aborted -r 1 [ 1159.681291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_aborted -r 1 [ 1159.723097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_aborted -r 1 [ 1159.751812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_aborted -r 1 [ 1159.803638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_aborted -r 1 [ 1159.829143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1159.847988] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1159.853251] LNet: Skipped 2 previous similar messages [ 1160.060422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.088286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.112581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.141200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.165948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.187386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.218213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.242346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.257227] LNet: 60819:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.270363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.294217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.321149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.357644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.381496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.411727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.441863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.950557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_no_route -r 1 [ 1160.998621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_no_route -r 1 [ 1161.033035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_no_route -r 1 [ 1161.064452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_no_route -r 1 [ 1161.102272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_no_route -r 1 [ 1161.135714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_no_route -r 1 [ 1161.169371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_no_route -r 1 [ 1161.200775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_no_route -r 1 [ 1161.231491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1161.434367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.463539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.490428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.516993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.548480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.581407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.606584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.639741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.672903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.717865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.746832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.782470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.841616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.876392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.923718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.430457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_timeout -r 1 [ 1162.477767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_timeout -r 1 [ 1162.512380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_timeout -r 1 [ 1162.565315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_timeout -r 1 [ 1162.628116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_timeout -r 1 [ 1162.660829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_timeout -r 1 [ 1162.694895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_timeout -r 1 [ 1162.748337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_timeout -r 1 [ 1162.785388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1162.807887] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1162.812919] LNet: Skipped 5 previous similar messages [ 1162.985744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.016811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.042774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.065956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.091414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.115745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.140428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.162623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.177936] LNet: 61061:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.185725] LNet: 61061:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1163.200221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.225462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.251550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.275815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.301761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.329569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.355803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.876245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_error -r 1 [ 1163.935166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_error -r 1 [ 1163.983904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_error -r 1 [ 1164.026959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_error -r 1 [ 1164.068993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_error -r 1 [ 1164.108340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_error -r 1 [ 1164.152975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_error -r 1 [ 1164.181266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_error -r 1 [ 1164.210780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1164.442673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.473922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.502299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.534936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.567819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.611545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.633873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.661495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.705865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.730870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.758332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.792578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.821557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.846357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.870599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.837465] LNet: 61263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.862427] LNetError: 61263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.918883] LNet: Removed LNI 192.168.203.23@tcp [ 1170.922152] LNetError: 61263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.946622] LNet: Removed LNI 192.168.203.23@tcp1 [ 1171.367802] Key type .llcrypt unregistered [ 1171.369923] Key type ._llcrypt unregistered [ 1191.160887] Key type ._llcrypt registered [ 1191.162578] Key type .llcrypt registered [ 1192.642499] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:02:15 (1786744935) [ 1193.524563] Key type .llcrypt unregistered [ 1193.529111] Key type ._llcrypt unregistered [ 1201.783653] Key type ._llcrypt registered [ 1201.795566] Key type .llcrypt registered [ 1202.025473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.037926] alg: No test for adler32 (adler32-zlib) [ 1203.014856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.051649] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1211.016530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1212.079927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_dropped -r 1 [ 1212.137556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_dropped -r 1 [ 1212.208217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1212.233183] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1212.498119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.535521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.585683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.625439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.689334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.719991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.750896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.789709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.807846] LNet: 62522:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.828841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.867824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.897664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.939585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.998923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.042933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.082847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.711785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_error -r 1 [ 1213.759655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_error -r 1 [ 1213.814387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1213.853980] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1214.073795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.095272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.115938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.134419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.156047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.177605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.198951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.220407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.234523] LNet: 62631:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.247227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.270108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.290874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.315053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.336973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.360484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.394911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.751449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_timeout -r 1 [ 1214.776541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_timeout -r 1 [ 1214.827471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1215.040186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.075888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.101577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.135406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.169520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.202956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.252486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.290835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.308478] LNet: 62741:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.321960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.353143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.379709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.413647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.439971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.476301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.516542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.888576] LNet: 62786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.893889] LNetError: 62786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.909130] LNet: Removed LNI 192.168.203.23@tcp [ 1216.264639] Key type .llcrypt unregistered [ 1216.269799] Key type ._llcrypt unregistered [ 1231.996196] Key type ._llcrypt registered [ 1231.998588] Key type .llcrypt registered [ 1233.242920] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:02:56 (1786744976) [ 1234.058241] Key type .llcrypt unregistered [ 1234.061396] Key type ._llcrypt unregistered [ 1239.609503] Key type ._llcrypt registered [ 1239.611246] Key type .llcrypt registered [ 1239.949115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.973451] alg: No test for adler32 (adler32-zlib) [ 1240.936397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.972896] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1249.975292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1260.527206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.566791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1260.604798] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1261.910288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_dropped -r 1 [ 1261.962982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e remote_dropped -r 1 [ 1262.012871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_dropped -r 1 [ 1262.071552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e remote_dropped -r 1 [ 1262.143165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e remote_dropped -r 1 [ 1262.232289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e remote_dropped -r 1 [ 1262.289438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e remote_dropped -r 1 [ 1262.353424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e remote_dropped -r 1 [ 1262.429965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1262.461967] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1262.475523] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.123@tcp [ 1262.818104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.859481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.894220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.937351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.994845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.033639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.084280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.134980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.151813] LNet: 64258:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.179664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.220538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.250142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.282436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.322944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.364461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.398199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.458586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.096457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_error -r 1 [ 1264.140425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e remote_error -r 1 [ 1264.164610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_error -r 1 [ 1264.233383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e remote_error -r 1 [ 1264.292446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e remote_error -r 1 [ 1264.339768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e remote_error -r 1 [ 1264.406789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e remote_error -r 1 [ 1264.453798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e remote_error -r 1 [ 1264.533895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1264.572183] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1264.577490] LNet: Skipped 2 previous similar messages [ 1264.923366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.952276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.980054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.013177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1265.054872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.097512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.138214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.174712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.195579] LNet: 64381:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.233941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.275874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.324961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.365446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.406875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.439428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.474418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.502457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.054798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_timeout -r 1 [ 1266.131123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e remote_timeout -r 1 [ 1266.197716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_timeout -r 1 [ 1266.243559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e remote_timeout -r 1 [ 1266.300383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e remote_timeout -r 1 [ 1266.352430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e remote_timeout -r 1 [ 1266.423966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e remote_timeout -r 1 [ 1266.468816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e remote_timeout -r 1 [ 1266.505575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.123@tcp [ 1266.533497] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1266.956833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.014800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.045476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.081769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.109881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.195848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.267509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.322593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.338232] LNet: 64505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1267.361494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.417540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.496413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.534430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.572593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.647227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.707544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.743866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.151764] LNet: 64587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.156903] LNetError: 64587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.163324] LNet: Removed LNI 192.168.203.23@tcp [ 1273.172229] LNet: Removed LNI 192.168.203.23@tcp1 [ 1273.473946] Key type .llcrypt unregistered [ 1273.475352] Key type ._llcrypt unregistered [ 1290.764418] Key type ._llcrypt registered [ 1290.771587] Key type .llcrypt registered [ 1291.994293] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:03:55 (1786745035) [ 1293.701875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1295.587089] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:03:58 (1786745038) [ 1296.796386] Key type .llcrypt unregistered [ 1296.798816] Key type ._llcrypt unregistered [ 1303.655451] Key type ._llcrypt registered [ 1303.664159] Key type .llcrypt registered [ 1304.047567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1304.085203] alg: No test for adler32 (adler32-zlib) [ 1305.122148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1305.144480] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1315.098435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1315.857052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e network_timeout -r 1 [ 1315.909240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e network_timeout -r 1 [ 1315.980537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1315.994123] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1316.307332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1316.353473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1316.392826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1316.437690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1316.447354] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1316.490783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1316.550801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1316.616161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1316.694312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1316.729224] LNet: 66151:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.780375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1316.842337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1316.897325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.946935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.051752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.134396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.798648] LNet: 66194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.804756] LNetError: 66194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.839388] LNet: Removed LNI 192.168.203.23@tcp [ 1318.282461] Key type .llcrypt unregistered [ 1318.284840] Key type ._llcrypt unregistered [ 1336.808121] Key type ._llcrypt registered [ 1336.810525] Key type .llcrypt registered [ 1336.969226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1336.980323] alg: No test for adler32 (adler32-zlib) [ 1337.881681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1337.908633] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1345.345896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1355.931570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.997171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1356.039939] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1357.678688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e network_timeout -r 1 [ 1357.727206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e network_timeout -r 1 [ 1357.793828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e network_timeout -r 1 [ 1357.857076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e network_timeout -r 1 [ 1357.931034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e network_timeout -r 1 [ 1358.005633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e network_timeout -r 1 [ 1358.094327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e network_timeout -r 1 [ 1358.149055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e network_timeout -r 1 [ 1358.229344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1358.276624] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1358.435137] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1358.439836] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.123@tcp [ 1358.579178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.626567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1358.703143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.760460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.834671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.927347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.000149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.071161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.104532] LNet: 67080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.156417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.206921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.257456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.314361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.364367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.423927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.463522] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1364.594114] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1364.958333] LNet: 67158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.970334] LNetError: 67158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.978980] LNet: Removed LNI 192.168.203.23@tcp [ 1365.000360] LNet: Removed LNI 192.168.203.23@tcp1 [ 1365.416643] Key type .llcrypt unregistered [ 1365.419151] Key type ._llcrypt unregistered [ 1382.242134] Key type ._llcrypt registered [ 1382.244562] Key type .llcrypt registered [ 1383.784353] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:05:26 (1786745126) [ 1384.158358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1384.169621] alg: No test for adler32 (adler32-zlib) [ 1385.108065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.155284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.171726] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1385.649663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1385.672792] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1386.303931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 1386.355488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1386.387727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1386.412832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1386.438625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1386.461447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1386.508611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.23@tcp [ 1386.535244] LNetError: 68260:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.23@tcp to recovery [ 1387.167603] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1387.172116] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1389.216526] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1390.007721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1393.311730] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1393.511443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1393.556513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1393.617750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.647569] LNet: 68367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.661985] LNetError: 68367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.673978] LNet: Removed LNI 192.168.203.23@tcp [ 1393.696467] LNet: Removed LNI 192.168.203.23@tcp1 [ 1393.713298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.796454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.827328] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1394.409465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1394.426635] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1395.161320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 1395.203391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1395.279282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1395.336463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1395.365713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1395.403766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1395.438771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1395.481894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.23@tcp [ 1395.501687] LNetError: 68640:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.23@tcp to recovery [ 1395.807412] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1399.095400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1401.951558] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1401.959819] LNet: Skipped 1 previous similar message [ 1405.964911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1406.000573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1406.029595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1411.355514] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:05:54 (1786745154) [ 1411.451071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.480329] LNet: 69347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.492296] LNetError: 69347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.505844] LNetError: 69347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.519211] LNet: Removed LNI 192.168.203.23@tcp [ 1411.532945] LNet: Removed LNI 192.168.203.23@tcp1 [ 1411.544923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.594400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.624159] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1412.169712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1412.188880] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1412.801120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 1412.860955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1412.885714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1412.917777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1412.951815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.991311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1413.030816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.23@tcp --health 0 [ 1413.120327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.599489] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.23@tcp [ 1414.623915] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1414.629861] LNet: Skipped 1 previous similar message [ 1429.469376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1429.622387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1429.681570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.727516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.23@tcp --health 500 [ 1429.858298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1436.526818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1436.582459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.598450] LNet: 69849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.603619] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.609149] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1436.615262] LNet: Removed LNI 192.168.203.23@tcp [ 1436.626864] LNet: Removed LNI 192.168.203.23@tcp1 [ 1436.642692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.681239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1436.707736] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1437.260853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1437.285710] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1438.063925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 1438.095552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1438.149107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1438.176890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1438.205055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1438.234993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1438.262180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1438.291371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.23@tcp --health 0 [ 1439.716397] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1439.727269] LNet: Skipped 3 previous similar messages [ 1440.652817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.795743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.819774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1450.848556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1456.118559] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:06:39 (1786745199) [ 1456.662446] LNet: 70906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.668732] LNetError: 70906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.676674] LNetError: 70906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1456.686176] LNet: Removed LNI 192.168.203.23@tcp [ 1456.707074] LNet: Removed LNI 192.168.203.23@tcp1 [ 1457.294315] Key type .llcrypt unregistered [ 1457.298493] Key type ._llcrypt unregistered [ 1463.254732] Key type ._llcrypt registered [ 1463.260314] Key type .llcrypt registered [ 1463.464521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.489160] alg: No test for adler32 (adler32-zlib) [ 1464.404607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1464.441419] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1472.715486] Lustre: DEBUG MARKER: Initial discovery [ 1472.741140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.123@tcp [ 1474.477855] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1474.790722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1474.854486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.123@tcp [ 1474.910122] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1474.941905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1475.023372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.123@tcp -r 1 -m GET -l 3 [ 1475.084237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.123@tcp -r 1 -m GET -e local_error [ 1475.138660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.123@tcp -r 1 -m PUT -l 6 [ 1475.744065] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1476.684604] Lustre: DEBUG MARKER: Force oleg323-client.virtnet to discover 192.168.203.123@tcp (in background) [ 1476.950740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.123@tcp [ 1478.294360] Lustre: DEBUG MARKER: Force oleg323-server to discover 192.168.203.23@tcp [ 1480.032235] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1480.295897] Lustre: DEBUG MARKER: Wait for 71674 [ 1481.655677] Lustre: DEBUG MARKER: Finished wait on 71674 [ 1481.901953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1481.935470] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1481.946213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1482.293796] LNet: 72127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.300992] LNetError: 72127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.367839] LNet: Removed LNI 192.168.203.23@tcp [ 1483.719982] Key type .llcrypt unregistered [ 1483.724237] Key type ._llcrypt unregistered [ 1499.099403] Key type ._llcrypt registered [ 1499.101341] Key type .llcrypt registered [ 1500.215584] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:07:23 (1786745243) [ 1500.937263] Key type .llcrypt unregistered [ 1500.941493] Key type ._llcrypt unregistered [ 1506.537626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1506.647933] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1506.667186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1506.674459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1509.161787] Key type ._llcrypt registered [ 1509.163518] Key type .llcrypt registered [ 1509.310739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1509.327397] alg: No test for adler32 (adler32-zlib) [ 1510.208624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.260344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.294791] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1510.977728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1510.995746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1518.859245] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:07:41 (1786745261) [ 1518.949650] LNet: 74202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.956386] LNetError: 74202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1518.967635] LNet: Removed LNI 10.1.2.3@tcp [ 1518.997223] LNet: Removed LNI 192.168.203.23@tcp [ 1519.593510] Key type .llcrypt unregistered [ 1519.599458] Key type ._llcrypt unregistered [ 1525.334197] Key type ._llcrypt registered [ 1525.336369] Key type .llcrypt registered [ 1525.472096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1525.488620] alg: No test for adler32 (adler32-zlib) [ 1526.465776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.535210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1526.568369] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1527.487289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1527.511686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1528.582674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.23@tcp [ 1529.631348] LNetError: Unexpected error -22 connecting to 192.168.203.23@tcp at host 192.168.203.23:988 [ 1534.605272] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:07:57 (1786745277) [ 1534.706319] LNet: 75335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.724284] LNetError: 75335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1534.732706] LNet: Removed LNI 10.1.2.3@tcp [ 1534.748722] LNet: Removed LNI 192.168.203.23@tcp [ 1535.367546] Key type .llcrypt unregistered [ 1535.375694] Key type ._llcrypt unregistered [ 1542.015639] Key type ._llcrypt registered [ 1542.022670] Key type .llcrypt registered [ 1542.315064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.328855] alg: No test for adler32 (adler32-zlib) [ 1543.248888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.294845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.324030] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1544.265506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1544.288678] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 1545.402860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.23@tcp1 --nid 192.168.203.23@tcp2 [ 1550.352151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp1 192.168.203.23@tcp2 [ 1550.386227] LNetError: 76184:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.23@tcp1 to 192.168.203.23@tcp2 - Route table may be misconfigured [ 1550.428480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp2 192.168.203.23@tcp1 [ 1559.367050] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:08:21 (1786745301) [ 1559.623241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.739222] LNet: 76753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.748933] LNetError: 76753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.770310] LNet: Removed LNI 192.168.203.23@tcp1 [ 1559.803348] LNet: Removed LNI 192.168.203.23@tcp2 [ 1559.915460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.989673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.009340] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1560.889798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1560.916584] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1561.594706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 1561.622486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.23@tcp [ 1561.647744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp1 192.168.203.23@tcp1 [ 1561.684985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.23@tcp -d 192.168.203.23@tcp -e network_timeout [ 1561.716866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -e network_timeout [ 1561.744280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -e network_timeout [ 1561.776097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -e network_timeout [ 1561.835825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1561.862365] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1561.893115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1562.015375] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1562.026672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.089247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1570.217931] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:08:32 (1786745312) [ 1570.271745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.304248] LNet: 77615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.309920] LNetError: 77615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.315963] LNetError: 77615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.322187] LNet: Removed LNI 192.168.203.23@tcp [ 1571.369933] LNet: Removed LNI 192.168.203.23@tcp1 [ 1571.408980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.502589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1571.920486] LNet: 77661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.336776] Key type .llcrypt unregistered [ 1572.340762] Key type ._llcrypt unregistered [ 1584.564569] Key type ._llcrypt registered [ 1584.568731] Key type .llcrypt registered [ 1585.812391] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:08:48 (1786745328) [ 1586.241702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.258191] alg: No test for adler32 (adler32-zlib) [ 1587.216488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.499811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1590.060190] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:08:53 (1786745333) [ 1590.210095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.229620] LNet: 78756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.267460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.336787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.373190] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1591.121955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1591.137829] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1591.921247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1591.962429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp1 [ 1592.023737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp1 [ 1597.845354] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:09:01 (1786745341) [ 1599.328274] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg323-server [ 1600.934348] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:09:04 (1786745344) [ 1602.541763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg323-server [ 1604.460723] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:09:07 (1786745347) [ 1606.228678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg323-server [ 1607.854540] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:09:10 (1786745350) [ 1609.438889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg323-server [ 1611.127102] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:09:14 (1786745354) [ 1613.141312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg323-server [ 1615.491329] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:09:18 (1786745358) [ 1617.533624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg323-server [ 1619.498796] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:09:22 (1786745362) [ 1621.102710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg323-server [ 1623.126325] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:09:26 (1786745366) [ 1624.925990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg323-server [ 1626.409782] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:09:29 (1786745369) [ 1628.013193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg323-server [ 1629.846607] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:09:32 (1786745372) [ 1630.183805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.229906] LNet: 82636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.236593] LNetError: 82636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.266225] LNet: Removed LNI 192.168.203.23@tcp [ 1631.267897] LNetError: 82636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.287260] LNet: Removed LNI 192.168.203.23@tcp1 [ 1631.332403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.371980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.388219] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1632.055369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1632.219280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1632.435872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.455695] LNet: 82783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.468172] LNetError: 82783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.492692] LNet: Removed LNI 192.168.203.23@tcp [ 1632.516644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.579990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1632.602624] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1633.267492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1633.472618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.537644] LNet: Removed LNI 192.168.203.23@tcp [ 1634.584907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.656553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.687259] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1635.501512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1635.563889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1635.751367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.779239] LNet: 83076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.786348] LNet: 83076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1635.797220] LNetError: 83076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.806440] LNetError: 83076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.825433] LNet: Removed LNI 192.168.203.23@tcp [ 1635.860101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.935763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1635.966877] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1636.823536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1637.131847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.245764] LNet: Removed LNI 192.168.203.23@tcp [ 1638.273411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.344957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.388705] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1639.330351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1639.401174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1639.580136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.700468] LNet: Removed LNI 192.168.203.23@tcp [ 1640.728083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.801107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1640.840115] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1641.783400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1641.993765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.020316] LNet: 83513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.025828] LNet: 83513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1642.033626] LNetError: 83513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.039663] LNetError: 83513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1642.051462] LNet: Removed LNI 192.168.203.23@tcp [ 1642.070980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.140824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.203139] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1643.008991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1643.082131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1643.321790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.392779] LNet: Removed LNI 192.168.203.23@tcp [ 1644.420754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.507173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1644.543796] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1645.186234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1645.423665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.507685] LNet: Removed LNI 192.168.203.23@tcp [ 1646.539146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.600640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.639625] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1647.454525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1647.562297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1647.988470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.071445] LNet: Removed LNI 192.168.203.23@tcp [ 1649.099910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.190339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1649.222976] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1649.896559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1650.167687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.213271] LNet: 84099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.221921] LNet: 84099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1650.230471] LNetError: 84099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.239982] LNetError: 84099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1650.251701] LNet: Removed LNI 192.168.203.23@tcp [ 1650.314936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.390131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.431061] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1651.307873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1651.397946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1652.944801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.004423] LNet: Removed LNI 192.168.203.23@tcp [ 1654.065509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.153407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1654.183905] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1655.191949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1655.465402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.556526] LNet: Removed LNI 192.168.203.23@tcp [ 1656.614422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.694433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.729607] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1657.418983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1657.475178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.499143] LNet: Removed LNI 192.168.203.23@tcp [ 1657.516230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.569514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.590978] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1658.410070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1658.505625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.548367] LNet: Removed LNI 192.168.203.23@tcp [ 1658.578969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.633630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1658.728953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.774552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.832420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1658.901389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.932198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.980910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1659.048787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.093906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.133724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1659.157575] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1659.770842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.797792] LNet: Removed LNI 192.168.203.23@tcp [ 1659.840469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.922896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1665.787614] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:10:09 (1786745409) [ 1665.880696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.914705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.955932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1666.015204] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/180] [ 1666.938419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.956204] LNet: 85559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.960188] LNet: 85559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1666.964891] LNetError: 85559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.969663] LNetError: 85559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1666.979782] LNet: Removed LNI 192.168.203.23@tcp231 [ 1666.992940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.039511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1667.056929] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/180] [ 1667.062579] LNet: 85567:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1667.408905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1667.458668] LNet: Removed LNI 192.168.203.23@tcp231 [ 1667.471608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1667.501582] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/180] [ 1668.574934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.620773] LNet: Removed LNI 192.168.203.23@tcp231 [ 1668.641396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.710787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1668.747565] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/0] [ 1668.753580] LNet: 85719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1669.188742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1669.233681] LNet: Removed LNI 192.168.203.23@tcp231 [ 1669.269868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1669.298313] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/0] [ 1670.055147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.075626] LNet: Removed LNI 192.168.203.23@tcp231 [ 1670.091200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.144967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1670.164708] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/60] [ 1670.168879] LNet: 85871:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1670.597779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1670.623976] LNet: Removed LNI 192.168.203.23@tcp231 [ 1670.636985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1670.660929] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/60] [ 1671.696766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.721669] LNet: Removed LNI 192.168.203.23@tcp231 [ 1671.737429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.783270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1671.803256] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/180] [ 1672.243712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1672.283778] LNet: Removed LNI 192.168.203.23@tcp231 [ 1672.307996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1672.336857] LNet: Added LNI 192.168.203.23@tcp231 [8/256/0/180] [ 1678.321603] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:10:21 (1786745421) [ 1678.506419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.532595] LNet: Removed LNI 192.168.203.23@tcp231 [ 1678.545171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.601179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1678.625501] LNet: Added LNI 192.168.203.23@tcp232 [8/256/0/180] [ 1679.283957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1679.341191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1679.369978] LNet: Removed LNI 192.168.203.23@tcp232 [ 1679.381772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1679.406158] LNet: Added LNI 192.168.203.23@tcp232 [8/256/0/180] [ 1680.205461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.232556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.255471] LNet: Removed LNI 192.168.203.23@tcp232 [ 1680.269678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1680.291951] LNet: Added LNI 192.168.203.23@tcp232 [8/256/0/180] [ 1681.034472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.078713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.110379] LNet: Removed LNI 192.168.203.23@tcp232 [ 1681.121985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1681.140544] LNet: Added LNI 192.168.203.23@tcp232 [8/256/0/180] [ 1681.775585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.806809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.834357] LNet: Removed LNI 192.168.203.23@tcp232 [ 1681.847427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1687.847694] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:10:31 (1786745431) [ 1689.108662] Key type .llcrypt unregistered [ 1689.111970] Key type ._llcrypt unregistered [ 1695.059982] Key type ._llcrypt registered [ 1695.070165] Key type .llcrypt registered [ 1695.322121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1695.339724] alg: No test for adler32 (adler32-zlib) [ 1696.234386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1696.258393] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1702.916969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1710.779418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.824708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.864199] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1712.210918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_interrupt -r 2 [ 1712.266191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_interrupt -r 2 [ 1712.310128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_interrupt -r 2 [ 1712.337620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_interrupt -r 2 [ 1712.373974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_interrupt -r 2 [ 1712.426919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_interrupt -r 2 [ 1712.450561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_interrupt -r 2 [ 1712.481644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_interrupt -r 2 [ 1712.517533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1712.535597] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1712.540583] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1712.737888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1712.774676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1712.804563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1712.827518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1712.853925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1712.884688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1712.911941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1712.946497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1712.961369] LNet: 88526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1712.976884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1713.000173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1713.024830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1713.049968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1713.072359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1713.106644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1713.556182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_dropped -r 2 [ 1713.584741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_dropped -r 2 [ 1713.624578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_dropped -r 2 [ 1713.673238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_dropped -r 2 [ 1713.721182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_dropped -r 2 [ 1713.767874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_dropped -r 2 [ 1713.836702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_dropped -r 2 [ 1713.890916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_dropped -r 2 [ 1713.932917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1713.954532] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1714.155628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1714.183798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1714.209978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1714.236407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1714.264977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1714.296473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.333691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.371580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.403288] LNet: 88646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.423311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.452595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.494297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.528658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.560444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.581729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.118173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_aborted -r 2 [ 1715.153557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_aborted -r 2 [ 1715.193458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_aborted -r 2 [ 1715.224900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_aborted -r 2 [ 1715.261893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_aborted -r 2 [ 1715.286762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_aborted -r 2 [ 1715.324285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_aborted -r 2 [ 1715.374325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_aborted -r 2 [ 1715.410846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1715.436503] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1715.703617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.748752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1715.785639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.806702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1715.830901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1715.863387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1715.898871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1715.928212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1715.947602] LNet: 88766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.977223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.018701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.067326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.095560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.128175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.176298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.705263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_no_route -r 2 [ 1716.752473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_no_route -r 2 [ 1716.783800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_no_route -r 2 [ 1716.821806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_no_route -r 2 [ 1716.850837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_no_route -r 2 [ 1716.876682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_no_route -r 2 [ 1716.905693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_no_route -r 2 [ 1716.936199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_no_route -r 2 [ 1716.983804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1717.277292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1717.303280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1717.330805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1717.382179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1717.414390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1717.460542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.512329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.569933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1717.608982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1717.650852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1717.702069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1717.741935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1717.768493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1717.797345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1718.253600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e local_timeout -r 2 [ 1718.283788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e local_timeout -r 2 [ 1718.314236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e local_timeout -r 2 [ 1718.342906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e local_timeout -r 2 [ 1718.381874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e local_timeout -r 2 [ 1718.418660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e local_timeout -r 2 [ 1718.446467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e local_timeout -r 2 [ 1718.487700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e local_timeout -r 2 [ 1718.531263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1718.555974] LNet: There was an unexpected network error while writing to 192.168.203.123: rc = -22 [ 1718.560877] LNet: Skipped 1 previous similar message [ 1718.818034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.850393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.881588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.914679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.941165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.988379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1719.053376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.087887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.099967] LNet: 89006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.104241] LNet: 89006:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1719.118731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.138868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.163192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.196335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.227636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.252831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.650803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp -m GET -e remote_dropped -r 2 [ 1719.684905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp -d 192.168.203.123@tcp1 -m GET -e remote_dropped -r 2 [ 1719.715919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp -m GET -e remote_dropped -r 2 [ 1719.751190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp -d 192.168.203.23@tcp1 -m GET -e remote_dropped -r 2 [ 1719.779914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp -m GET -e remote_dropped -r 2 [ 1719.806823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.123@tcp1 -m GET -e remote_dropped -r 2 [ 1719.836375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp -m GET -e remote_dropped -r 2 [ 1719.864356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.23@tcp1 -d 192.168.203.23@tcp1 -m GET -e remote_dropped -r 2 [ 1719.900149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.23@tcp 192.168.203.123@tcp [ 1719.936546] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.123@tcp [ 1720.149177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.180596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.207907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.236329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.263866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.311368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.350576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.387632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.423655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.455636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.484126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.513440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.540880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.574364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1725.210715] LNet: 89204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.221305] LNetError: 89204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.234746] LNet: Removed LNI 192.168.203.23@tcp [ 1725.259370] LNet: Removed LNI 192.168.203.23@tcp1 [ 1725.512732] Key type .llcrypt unregistered [ 1725.514324] Key type ._llcrypt unregistered [ 1740.393207] Key type ._llcrypt registered [ 1740.404351] Key type .llcrypt registered [ 1741.541763] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:11:24 (1786745484) [ 1742.039120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.054431] alg: No test for adler32 (adler32-zlib) [ 1742.948600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.993418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.016231] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1743.732662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1743.769282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.23@tcp [ 1743.789206] LNetError: 90200:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.23@tcp to recovery [ 1743.886715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.23@tcp [ 1744.042286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1749.809549] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:11:33 (1786745493) [ 1749.879679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.922733] LNet: 90784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.932176] LNetError: 90784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.956665] LNet: Removed LNI 192.168.203.23@tcp [ 1749.999055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.045357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.063451] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1750.829739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1750.844948] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 1751.686926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.23@tcp [ 1751.706266] LNetError: 91056:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.23@tcp to recovery [ 1752.095909] LNet: 1 local NIs in recovery (showing 1): 192.168.203.23@tcp [ 1752.105563] LNet: There was an unexpected network error while writing to 192.168.203.23: rc = -22 [ 1752.944603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.23@tcp [ 1760.484626] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:11:43 (1786745503) [ 1762.510290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg323-server [ 1764.542641] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:11:47 (1786745507) [ 1766.053039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg323-server [ 1767.544710] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:11:50 (1786745510) [ 1769.459263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg323-server [ 1771.177718] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:11:54 (1786745514) [ 1771.393708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.413108] LNet: 92681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.419929] LNetError: 92681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.424405] LNet: Removed LNI 192.168.203.23@tcp [ 1771.436879] LNet: Removed LNI 192.168.203.23@tcp1 [ 1771.464648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.558786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1771.618420] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1772.548858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1772.645350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1772.678888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1772.727133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1772.763754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1772.817492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1779.666711] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:12:02 (1786745522) [ 1779.756912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.782760] LNet: 93403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.791796] LNetError: 93403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.799763] LNetError: 93403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1779.830723] LNet: Removed LNI 192.168.203.23@tcp [ 1779.854031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.917987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.932721] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1786.207508] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:12:09 (1786745529) [ 1786.296125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.333624] LNet: 94108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.340753] LNetError: 94108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.361664] LNet: Removed LNI 192.168.203.23@tcp [ 1786.391674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.457912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.475227] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1793.588920] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:12:16 (1786745536) [ 1795.191727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1797.260644] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:12:20 (1786745540) [ 1797.694921] LNet: 95149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.701296] LNetError: 95149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.714854] LNet: Removed LNI 192.168.203.23@tcp [ 1798.456661] Key type .llcrypt unregistered [ 1798.459783] Key type ._llcrypt unregistered [ 1804.243777] Key type ._llcrypt registered [ 1804.246754] Key type .llcrypt registered [ 1804.463770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.498146] alg: No test for adler32 (adler32-zlib) [ 1805.465172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1805.519864] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1813.038765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 1817.624798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.123@tcp [ 1817.942642] LNet: 95575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.947902] LNetError: 95575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.962950] LNet: Removed LNI 192.168.203.23@tcp [ 1818.222295] Key type .llcrypt unregistered [ 1818.224499] Key type ._llcrypt unregistered [ 1829.953352] Key type ._llcrypt registered [ 1829.956215] Key type .llcrypt registered [ 1831.159104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1832.964651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1834.878277] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:12:57 (1786745577) [ 1836.304774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1837.925155] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:13:01 (1786745581) [ 1840.048737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg323-server [ 1841.945834] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:13:04 (1786745584) [ 1844.443978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg323-server [ 1846.744265] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:13:09 (1786745589) [ 1848.821463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.113489] alg: No test for adler32 (adler32-zlib) [ 1850.408601] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 1850.990224] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1852.967166] Key type lgssc registered [ 1855.093363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.047837] Key type lgssc unregistered [ 1883.304493] LNet: 98755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.318043] LNetError: 98755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.341564] LNet: Removed LNI 192.168.203.23@tcp [ 1884.149644] Key type .llcrypt unregistered [ 1884.151478] Key type ._llcrypt unregistered [ 1898.877502] Key type ._llcrypt registered [ 1898.880537] Key type .llcrypt registered [ 1900.137625] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:14:03 (1786745643) [ 1900.510858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.523529] alg: No test for adler32 (adler32-zlib) [ 1901.497525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.599494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.669032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.714874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.763682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.946591] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:14:11 (1786745651) [ 1909.094590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.131399] LNet: 100167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.187769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.269345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.293050] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1910.082806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.104979] LNet: 100315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.113413] LNetError: 100315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.124607] LNet: Removed LNI 192.168.203.23@tcp [ 1910.144130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.210248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1910.235103] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1910.788867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.827039] LNetError: 100344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.842890] LNet: Removed LNI 192.168.203.23@tcp [ 1910.878848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.999679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1911.058920] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1917.524766] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:14:20 (1786745660) [ 1917.752129] LNet: 100934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.761524] LNet: 100934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1917.769529] LNetError: 100934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.786651] LNet: Removed LNI 192.168.203.23@tcp [ 1918.286573] Key type .llcrypt unregistered [ 1918.288829] Key type ._llcrypt unregistered [ 1924.301982] Key type ._llcrypt registered [ 1924.305975] Key type .llcrypt registered [ 1924.532757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1924.553853] alg: No test for adler32 (adler32-zlib) [ 1925.442800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.533182] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1925.536931] LNet: *** cfs_fail_loc=e003, val=100*** [ 1926.167398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1926.190363] LNet: *** cfs_fail_loc=e003, val=100*** [ 1926.198023] LNet: Skipped 2 previous similar messages [ 1926.210682] Ping reply received with mapping: [ 1926.213192] found metadata reply NI[192.168.203.23@tcp] [ 1926.217810] 17 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1926.224975] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1926.228870] 5a 5a 5a 5a 5a 5a 5a 5a [ 1926.354387] LNet: 101336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.369886] LNetError: 101336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.386034] LNet: Removed LNI 192.168.203.23@tcp [ 1927.152767] Key type .llcrypt unregistered [ 1927.161634] Key type ._llcrypt unregistered [ 1939.279431] Key type ._llcrypt registered [ 1939.281914] Key type .llcrypt registered [ 1940.365615] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:14:43 (1786745683) [ 1940.943420] Key type .llcrypt unregistered [ 1940.948457] Key type ._llcrypt unregistered [ 1947.191692] Key type ._llcrypt registered [ 1947.199557] Key type .llcrypt registered [ 1947.382853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1947.399339] alg: No test for adler32 (adler32-zlib) [ 1948.516097] LNetError: 102394:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1948.899479] Key type .llcrypt unregistered [ 1948.901954] Key type ._llcrypt unregistered [ 1954.276165] Key type ._llcrypt registered [ 1954.279951] Key type .llcrypt registered [ 1955.745032] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:14:58 (1786745698) [ 1956.509189] Key type .llcrypt unregistered [ 1956.511126] Key type ._llcrypt unregistered [ 1963.993267] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1964.002232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1965.610506] Key type ._llcrypt registered [ 1965.612233] Key type .llcrypt registered [ 1965.808039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1965.822232] alg: No test for adler32 (adler32-zlib) [ 1966.802150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.871881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1977.514437] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:15:20 (1786745720) [ 1977.724696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.750327] LNet: 104173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.760923] LNetError: 104173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1977.791627] LNet: Removed LNI 10.1.2.3@tcp [ 1977.837552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.493754] Key type .llcrypt unregistered [ 1978.498333] Key type ._llcrypt unregistered [ 1985.772332] Key type ._llcrypt registered [ 1985.784756] Key type .llcrypt registered [ 1986.039938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.086149] alg: No test for adler32 (adler32-zlib) [ 1986.992609] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 1987.074792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 1987.281804] LNet: 104468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.295590] LNetError: 104468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.315838] LNet: Removed LNI 192.168.203.23@tcp [ 1988.080517] Key type .llcrypt unregistered [ 1988.090972] Key type ._llcrypt unregistered [ 1999.440916] Key type ._llcrypt registered [ 1999.443420] Key type .llcrypt registered [ 2000.677914] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:15:43 (1786745743) [ 2001.273902] Key type .llcrypt unregistered [ 2001.284313] Key type ._llcrypt unregistered [ 2007.514674] Key type ._llcrypt registered [ 2007.516956] Key type .llcrypt registered [ 2007.688651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.703851] alg: No test for adler32 (adler32-zlib) [ 2014.624341] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:15:57 (1786745757) [ 2014.710831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.774352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.863546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2014.935783] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2015.672209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.694431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2015.713865] LNet: Removed LNI 192.168.203.23@tcp [ 2015.885959] LNet: 106225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.145586] Key type .llcrypt unregistered [ 2016.148126] Key type ._llcrypt unregistered [ 2026.108711] Key type ._llcrypt registered [ 2026.111729] Key type .llcrypt registered [ 2027.653641] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:16:10 (1786745770) [ 2028.056388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.087289] alg: No test for adler32 (adler32-zlib) [ 2029.050419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.128042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.161113] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2029.832221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2029.895318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.23@tcp [ 2029.928845] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2029.931782] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2029.934974] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2029, 0, 0 [ 2029.939657] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2029.942584] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2029.946354] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2029.950424] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2029, 0, 0 [ 2029.954593] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2029.956940] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2029.960702] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2029.969494] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2029, 0, 2079 [ 2029.974224] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2029.976546] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2029.980919] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2029.986132] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2029, 0, 2079 [ 2029.991475] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2035.626456] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:16:18 (1786745778) [ 2036.294440] LNet: 107765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.300546] LNetError: 107765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.316034] LNet: Removed LNI 192.168.203.23@tcp [ 2036.987335] Key type .llcrypt unregistered [ 2036.991199] Key type ._llcrypt unregistered [ 2043.279426] Key type ._llcrypt registered [ 2043.281550] Key type .llcrypt registered [ 2043.428875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2043.441515] alg: No test for adler32 (adler32-zlib) [ 2044.460723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2044.491908] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2052.262171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 2061.203145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.272475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2061.328712] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 2062.282433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2062.541691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2062.551244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2062.556390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2064.881294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2064.896034] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2075.923512] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:16:58 (1786745818) [ 2076.327395] LNet: 109279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.338506] LNetError: 109279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.348841] LNet: Removed LNI 192.168.203.23@tcp [ 2077.407253] LNet: Removed LNI 192.168.203.23@tcp1 [ 2077.413712] LNetError: 109279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2077.421236] LNetError: 109279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2077.444914] LNet: Removed LNI 10.1.2.3@tcp99 [ 2078.315053] Key type .llcrypt unregistered [ 2078.318581] Key type ._llcrypt unregistered [ 2084.945215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2084.956048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2087.571788] Key type ._llcrypt registered [ 2087.578345] Key type .llcrypt registered [ 2087.812622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2087.827816] alg: No test for adler32 (adler32-zlib) [ 2088.796577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.899501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.938481] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2089.791740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2089.830115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2091.747756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2091.799889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2091.853678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2091.883619] LNetError: 110027:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2091.920317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2091.984730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2092.460116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2092.594498] LNet: 110047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.604664] LNetError: 110047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2092.615184] LNet: Removed LNI 10.1.2.3@tcp [ 2093.678647] LNet: Removed LNI 192.168.203.23@tcp [ 2094.470569] Key type .llcrypt unregistered [ 2094.479805] Key type ._llcrypt unregistered [ 2106.275850] Key type ._llcrypt registered [ 2106.281155] Key type .llcrypt registered [ 2107.466949] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:17:30 (1786745850) [ 2107.865757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.877879] alg: No test for adler32 (adler32-zlib) [ 2108.807572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.872827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.905804] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2109.528898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2109.605584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg323-client@tcp [ 2115.819081] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:17:38 (1786745858) [ 2115.948529] LNet: 111578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.953401] LNetError: 111578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.979917] LNet: Removed LNI 192.168.203.23@tcp [ 2116.600577] Key type .llcrypt unregistered [ 2116.603887] Key type ._llcrypt unregistered [ 2122.780321] Key type ._llcrypt registered [ 2122.783543] Key type .llcrypt registered [ 2122.943631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.971429] alg: No test for adler32 (adler32-zlib) [ 2123.914587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2123.961960] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2124.003773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2129.980858] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:17:52 (1786745872) [ 2131.500377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2133.461558] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:17:56 (1786745876) [ 2135.175198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2136.993612] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:17:59 (1786745879) [ 2137.135906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.170239] LNet: 113001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.178054] LNetError: 113001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.188820] LNet: Removed LNI 192.168.203.23@tcp [ 2137.220271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.296641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.325031] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2137.918856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2137.957339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2144.085111] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:18:07 (1786745887) [ 2144.230861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.276687] LNet: 113711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.294052] LNetError: 113711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.318679] LNet: Removed LNI 192.168.203.23@tcp [ 2144.360154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.441484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2144.480689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2145.246489] Key type .llcrypt unregistered [ 2145.249586] Key type ._llcrypt unregistered [ 2157.739156] Key type ._llcrypt registered [ 2157.746792] Key type .llcrypt registered [ 2159.375589] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:18:21 (1786745901) [ 2160.027746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.039996] alg: No test for adler32 (adler32-zlib) [ 2161.047183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.158191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.197074] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2161.971180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2162.044476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2170.569402] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:18:33 (1786745913) [ 2170.693106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.739467] LNet: 115270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.755080] LNetError: 115270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.785038] LNet: Removed LNI 192.168.203.23@tcp [ 2170.814750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.869751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2170.900811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2178.146429] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:18:40 (1786745920) [ 2178.481513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.543866] LNet: 115857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.571537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.663474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2179.334427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2179.391133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2179.498541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2180.076281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2180.139292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2180.219891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.245924] LNet: 115890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.280336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.362224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2180.760613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2180.791170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2180.850887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2181.259027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2181.294548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2187.011232] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:18:50 (1786745930) [ 2187.158514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.182204] LNet: 116490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.205355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.270491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.308761] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2187.815560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2187.829300] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2188.535825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2188.577169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2188.753273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.769050] LNetError: 116770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.774882] LNet: Removed LNI 192.168.203.23@tcp [ 2188.784972] LNet: Removed LNI 192.168.203.23@tcp2 [ 2188.802622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.851889] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2188.857973] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2188.862781] LNet: 116776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2189.232459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2189.409538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.431171] LNet: Removed LNI 192.168.203.23@tcp [ 2190.436193] LNetError: 116813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.446197] LNetError: 116813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2190.469813] LNet: Removed LNI 192.168.203.23@tcp2 [ 2190.506595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.563158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.604086] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2191.257141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2191.275335] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2192.035614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2192.086750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2192.132783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2192.352908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.368810] LNet: 117097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.374820] LNet: 117097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2193.440032] LNet: Removed LNI 192.168.203.23@tcp [ 2193.442946] LNetError: 117097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.448745] LNetError: 117097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2193.464594] LNet: Removed LNI 192.168.203.23@tcp2 [ 2193.502975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.598603] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2193.607242] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2193.612513] LNet: 117104:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2194.065043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2194.270320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.291242] LNet: Removed LNI 192.168.203.23@tcp [ 2195.312366] LNet: Removed LNI 192.168.203.23@tcp2 [ 2195.357644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.420186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2195.450991] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2196.216780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2196.238906] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2197.075972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2197.116656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2197.420839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.495724] LNet: Removed LNI 192.168.203.23@tcp [ 2198.501282] LNetError: 117420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.512155] LNetError: 117420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2198.527835] LNet: Removed LNI 192.168.203.23@tcp2 [ 2198.577974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.672126] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2198.683054] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2198.696309] LNet: 117426:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.291626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2199.493437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.543210] LNet: Removed LNI 192.168.203.23@tcp [ 2200.569505] LNet: Removed LNI 192.168.203.23@tcp2 [ 2200.628345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.684540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2200.725364] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2201.472768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2201.489767] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2202.450348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2202.489401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2202.562640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2202.809979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.830388] LNet: 117748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.836194] LNet: 117748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2203.871497] LNet: Removed LNI 192.168.203.23@tcp [ 2204.901955] LNet: Removed LNI 192.168.203.23@tcp2 [ 2204.938915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.098069] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2205.110295] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2205.117681] LNet: 117754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2205.791076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2206.023206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.053649] LNet: Removed LNI 192.168.203.23@tcp [ 2207.078737] LNet: Removed LNI 192.168.203.23@tcp2 [ 2207.152166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.217248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.250808] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2207.843413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2207.869679] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2208.663444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2208.687449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2208.744720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2209.074576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.092084] LNetError: 118088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.095499] LNetError: 118088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2210.143166] LNet: Removed LNI 192.168.203.23@tcp [ 2210.154620] LNet: Removed LNI 192.168.203.23@tcp2 [ 2210.190884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.276334] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2210.282202] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2210.289552] LNet: 118094:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2210.751513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2211.072059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.127280] LNet: Removed LNI 192.168.203.23@tcp [ 2212.137991] LNet: Removed LNI 192.168.203.23@tcp2 [ 2212.162580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.255641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.284063] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2213.020904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.038577] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2213.832526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2213.884304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2213.919488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2213.997181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2214.363732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.391279] LNet: Removed LNI 192.168.203.23@tcp [ 2216.421697] LNet: Removed LNI 192.168.203.23@tcp2 [ 2216.489920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.586556] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2216.595813] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2217.296543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2224.779882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2226.492856] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:19:29 (1786745969) [ 2226.603466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.636073] LNet: 119211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.643221] LNet: 119211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2226.654378] LNetError: 119211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.665330] LNetError: 119211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2226.675704] LNet: Removed LNI 192.168.203.23@tcp [ 2227.752410] LNet: Removed LNI 192.168.203.23@tcp2 [ 2227.792917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.081748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.126717] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2228.873359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2228.891037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.751464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.777967] LNet: Removed LNI 10.1.2.3@tcp [ 2229.789415] LNet: Removed LNI 192.168.203.23@tcp [ 2229.809442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.867692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2229.886826] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2229.891929] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2230.433477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2230.474604] LNet: Removed LNI 192.168.203.23@tcp [ 2230.491976] LNet: Removed LNI 10.1.2.3@tcp [ 2237.029737] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:19:40 (1786745980) [ 2237.163854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.200700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.272431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.317899] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2237.948349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2237.984975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.026699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.052762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.083232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.109652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.135833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.164299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.193353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.231861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.294306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.23@tcp [ 2238.444764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2243.641721] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:19:46 (1786745986) [ 2243.830640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.902312] LNet: Removed LNI 192.168.203.23@tcp [ 2244.937760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.084456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2245.295885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2245.300624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2245.318268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2247.605300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2247.620484] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2248.433500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2248.452805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.506890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2250.565939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2250.662581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.23@tcp -m PUT -e local_timeout -r 1 [ 2250.747584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2250.838198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2250.844166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2251.199611] LNetError: 120892:0:(lib-move.c:4801:lnet_parse()) 192.168.203.23@tcp, src 192.168.203.23@tcp: Dropping PUT (error -108 looking up sender) [ 2252.263891] LNet: Removed LNI 10.1.2.3@tcp [ 2253.286157] LNet: Removed LNI 192.168.203.23@tcp [ 2253.652264] Key type .llcrypt unregistered [ 2253.655190] Key type ._llcrypt unregistered [ 2265.305888] Key type ._llcrypt registered [ 2265.307875] Key type .llcrypt registered [ 2266.721495] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:20:09 (1786746009) [ 2267.097846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.114179] alg: No test for adler32 (adler32-zlib) [ 2268.025879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.326656] LNet: 122207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.841593] Key type .llcrypt unregistered [ 2268.849681] Key type ._llcrypt unregistered [ 2275.095312] Key type ._llcrypt registered [ 2275.099651] Key type .llcrypt registered [ 2275.271612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2275.288441] alg: No test for adler32 (adler32-zlib) [ 2276.301588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2276.363213] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2283.094612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 2283.265623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2283.706580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.123@tcp [ 2283.976930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.004483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2284.034738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2284.068379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2284.129921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2284.208845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2284.272326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2284.317136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2284.351496] LNet: 122660:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2284.365530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2284.394854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2284.432288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2284.474822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2284.518699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2284.554949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2285.336102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2285.600928] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.123@tcp [ 2285.611831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2285.647835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2285.701149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2285.765168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2285.852787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2285.897137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2285.939517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2285.999241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.027816] LNet: 122777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.043573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.083143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.129536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2286.184886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2286.246659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2286.295330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.873957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.123@tcp [ 2287.093486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2287.117492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2287.143935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2287.189119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2287.217383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2287.246355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2287.273322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2287.333634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2287.366340] LNet: 122895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2287.395915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2287.429265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2287.458869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2287.494919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2287.522668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.557764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2288.192239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2288.425106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.474250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.504677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.568658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.620604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.662371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.730318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.761298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.808937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.859597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.903246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.934507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.955614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.993576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.384141] LNet: 123062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.393161] LNetError: 123062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2289.430235] LNet: Removed LNI 192.168.203.23@tcp [ 2289.777099] Key type .llcrypt unregistered [ 2289.779380] Key type ._llcrypt unregistered [ 2307.762869] Key type ._llcrypt registered [ 2307.765500] Key type .llcrypt registered [ 2309.075274] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:20:52 (1786746052) [ 2309.533684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.543485] alg: No test for adler32 (adler32-zlib) [ 2310.522826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.855949] LNet: 123964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.446411] Key type .llcrypt unregistered [ 2311.448364] Key type ._llcrypt unregistered [ 2319.545722] Key type ._llcrypt registered [ 2319.547425] Key type .llcrypt registered [ 2319.899463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.932296] alg: No test for adler32 (adler32-zlib) [ 2320.876233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2320.898870] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2327.895670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.123@tcp [ 2337.104575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.167477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2337.212966] LNet: Added LNI 192.168.203.23@tcp1 [8/256/0/180] [ 2338.223449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2338.892615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.123@tcp [ 2339.094495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2339.126807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2339.157269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2339.193586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2339.225200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2339.257796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2339.290267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2339.335522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2339.359603] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.123@tcp [ 2339.380106] LNet: 124621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2339.406957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2339.449817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2339.497606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2339.565779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2339.633290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2339.702099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2340.415483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2340.665638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2340.712579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2340.757414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2340.807776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2340.843838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2340.878540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2340.908808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2340.940521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2340.960651] LNet: 124738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.981231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2341.027258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2341.072953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2341.100973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2341.142808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2341.208942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2341.960734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.123@tcp [ 2342.254671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2342.292728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2342.330747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2342.386133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2342.427395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2342.453952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2342.508838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2342.592508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2342.621951] LNet: 124855:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2342.648252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2342.693186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2342.768182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2342.831710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2342.877550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2342.927628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2343.527805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2343.691857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2343.712873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2343.736162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2343.760561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2343.784917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2343.811942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2343.836162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2343.858265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2343.884168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2343.906552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2343.927719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2343.953854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2343.975159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2343.997916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2348.269515] LNet: 125057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.275925] LNetError: 125057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.283526] LNet: Removed LNI 192.168.203.23@tcp [ 2348.301902] LNet: Removed LNI 192.168.203.23@tcp1 [ 2348.629722] Key type .llcrypt unregistered [ 2348.631995] Key type ._llcrypt unregistered [ 2364.198821] Key type ._llcrypt registered [ 2364.204931] Key type .llcrypt registered [ 2365.351644] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:21:48 (1786746108) [ 2365.931442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.945636] alg: No test for adler32 (adler32-zlib) [ 2366.874894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.951611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.009103] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2367.946294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2367.967556] LNet: Added LNI 192.168.203.23@tcp2 [8/256/0/180] [ 2368.825341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.23@tcp [ 2368.925822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2369.952418] LNet: Removed LNI 192.168.203.23@tcp [ 2377.423535] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:22:00 (1786746120) [ 2378.936252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2380.667470] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:22:03 (1786746123) [ 2389.277033] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:22:11 (1786746131) [ 2397.778791] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:22:20 (1786746140) [ 2407.047756] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:22:29 (1786746149) [ 2415.286423] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:22:38 (1786746158) [ 2421.920313] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:22:45 (1786746165) [ 2428.773464] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:22:51 (1786746171) [ 2436.264872] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:22:59 (1786746179) [ 2444.431514] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:23:07 (1786746187) [ 2451.522577] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:23:14 (1786746194) [ 2459.118808] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:23:21 (1786746201) [ 2467.172790] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:23:29 (1786746209) [ 2479.546081] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:23:41 (1786746221) [ 2488.334503] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:23:51 (1786746231) [ 2497.095224] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:23:59 (1786746239) [ 2506.666869] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:24:09 (1786746249) [ 2513.329974] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:24:16 (1786746256) [ 2521.007461] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:24:24 (1786746264) [ 2521.241412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.265475] LNet: 136764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.270302] LNetError: 136764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.286597] LNet: Removed LNI 192.168.203.23@tcp2 [ 2521.335443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2521.535403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2521.567085] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2522.889941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2522.935782] LNet: 136906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.947402] LNetError: 136906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.969072] LNet: Removed LNI 192.168.203.23@tcp [ 2522.996925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2523.055615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2523.076915] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2530.940930] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:24:33 (1786746273) [ 2539.894993] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:24:43 (1786746283) [ 2547.508269] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:24:50 (1786746290) [ 2556.048901] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:24:58 (1786746298) [ 2562.975736] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:25:05 (1786746305) [ 2569.713702] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:25:12 (1786746312) [ 2577.705205] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:25:20 (1786746320) [ 2585.704499] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:25:28 (1786746328) [ 2593.882706] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:25:36 (1786746336) [ 2600.328811] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:25:43 (1786746343) [ 2607.499748] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:25:50 (1786746350) [ 2616.215199] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:25:58 (1786746358) [ 2623.579105] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2333 sec ========== 18:26:06 (1786746366) [ 2624.036965] LNet: 144372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.042667] LNetError: 144372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.075113] LNet: Removed LNI 192.168.203.23@tcp [ 2624.783801] Key type .llcrypt unregistered [ 2624.790942] Key type ._llcrypt unregistered [ 2638.854666] Key type ._llcrypt registered [ 2638.865978] Key type .llcrypt registered [ 2639.754487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2639.797678] alg: No test for adler32 (adler32-zlib) [ 2641.075651] Lustre: Lustre: Build Version: 2.17.57_1_g59aa375 [ 2641.457631] LNet: Added LNI 192.168.203.23@tcp [8/256/0/180] [ 2643.255506] Key type lgssc registered [ 2644.726374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2716.007585] Lustre: Mounted lustre-client - version 2.17.57_1_g59aa375 [ 2721.960516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2731.040672] Lustre: Unmounted lustre-client [ 2759.099073] Key type lgssc unregistered [ 2759.357140] LNet: 147333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2759.364886] LNetError: 147333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2759.389413] LNet: Removed LNI 192.168.203.23@tcp [ 2760.289133] Key type .llcrypt unregistered [ 2760.292024] Key type ._llcrypt unregistered