[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 352873075 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2608MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002222] x2apic enabled [ 0.003003] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.005935] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.006008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008012] LSM: Security Framework initializing [ 0.008924] Yama: becoming mindful. [ 0.009017] SELinux: Initializing. [ 0.010037] *** VALIDATE selinux *** [ 0.016622] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020292] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.021088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022059] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023046] *** VALIDATE tmpfs *** [ 0.024133] *** VALIDATE proc *** [ 0.024942] *** VALIDATE cgroup *** [ 0.025004] *** VALIDATE cgroup2 *** [ 0.026163] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028025] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030016] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.033527] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.035816] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036402] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037010] ... version: 2 [ 0.037828] ... bit width: 48 [ 0.038005] ... generic registers: 4 [ 0.038805] ... value mask: 0000ffffffffffff [ 0.039005] ... max period: 00007fffffffffff [ 0.040005] ... fixed-purpose events: 3 [ 0.040831] ... event mask: 000000070000000f [ 0.041102] rcu: Hierarchical SRCU implementation. [ 0.043010] smp: Bringing up secondary CPUs ... [ 0.044259] x86: Booting SMP configuration: [ 0.045014] .... node #0, CPUs: #1 #2 #3 [ 0.047534] smp: Brought up 1 node, 4 CPUs [ 0.048827] smpboot: Max logical packages: 1 [ 0.049006] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.229499] node 0 deferred pages initialised in 179ms [ 0.231494] devtmpfs: initialized [ 0.232148] x86/mm: Memory block size: 128MB [ 0.234157] gcov: version magic: 0x41383552 [ 0.235538] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.237066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239204] pinctrl core: initialized pinctrl subsystem [ 0.240070] [ 0.240370] ************************************************************* [ 0.242006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243005] ** ** [ 0.244005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245009] ** ** [ 0.247005] ** This means that this kernel is built to expose internal ** [ 0.248004] ** IOMMU data structures, which may compromise security on ** [ 0.249005] ** your system. ** [ 0.251006] ** ** [ 0.252004] ** If you see this message and you are not debugging the ** [ 0.253005] ** kernel, report this immediately to your vendor! ** [ 0.254006] ** ** [ 0.256005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257004] ************************************************************* [ 0.258475] NET: Registered protocol family 16 [ 0.260277] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265257] cpuidle: using governor menu [ 0.266294] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267411] PCI: Using configuration type 1 for base access [ 0.269090] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.275054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.278062] cryptd: max_cpu_qlen set to 1000 [ 0.280150] ACPI: Added _OSI(Module Device) [ 0.281011] ACPI: Added _OSI(Processor Device) [ 0.281973] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285987] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.289401] ACPI: Interpreter enabled [ 0.290041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290809] ACPI: Using IOAPIC for interrupt routing [ 0.292066] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293246] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.299859] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.301019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.303009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.304031] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.306868] acpiphp: Slot [2] registered [ 0.308075] acpiphp: Slot [5] registered [ 0.308878] acpiphp: Slot [6] registered [ 0.310060] acpiphp: Slot [3] registered [ 0.310830] acpiphp: Slot [4] registered [ 0.311049] acpiphp: Slot [7] registered [ 0.311827] acpiphp: Slot [8] registered [ 0.312043] acpiphp: Slot [9] registered [ 0.312842] acpiphp: Slot [10] registered [ 0.314058] acpiphp: Slot [11] registered [ 0.314852] acpiphp: Slot [12] registered [ 0.316043] acpiphp: Slot [13] registered [ 0.316839] acpiphp: Slot [14] registered [ 0.317041] acpiphp: Slot [15] registered [ 0.317737] acpiphp: Slot [16] registered [ 0.319051] acpiphp: Slot [17] registered [ 0.319866] acpiphp: Slot [18] registered [ 0.320042] acpiphp: Slot [19] registered [ 0.320806] acpiphp: Slot [20] registered [ 0.322042] acpiphp: Slot [21] registered [ 0.322803] acpiphp: Slot [22] registered [ 0.323058] acpiphp: Slot [23] registered [ 0.323844] acpiphp: Slot [24] registered [ 0.325046] acpiphp: Slot [25] registered [ 0.325836] acpiphp: Slot [26] registered [ 0.326043] acpiphp: Slot [27] registered [ 0.326840] acpiphp: Slot [28] registered [ 0.328056] acpiphp: Slot [29] registered [ 0.328856] acpiphp: Slot [30] registered [ 0.329043] acpiphp: Slot [31] registered [ 0.329818] PCI host bridge to bus 0000:00 [ 0.331011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339692] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.341837] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346399] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.349437] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354530] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356517] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359542] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.369007] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.371007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375017] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378007] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386797] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.391856] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.395007] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.398007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.404007] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.410469] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.411190] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.413184] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.414214] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.416102] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.419165] iommu: Default domain type: Passthrough [ 0.420236] SCSI subsystem initialized [ 0.421073] ACPI: bus type USB registered [ 0.422064] usbcore: registered new interface driver usbfs [ 0.423030] usbcore: registered new interface driver hub [ 0.424029] usbcore: registered new device driver usb [ 0.426103] pps_core: LinuxPPS API ver. 1 registered [ 0.427005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.429016] PTP clock support registered [ 0.430143] EDAC MC: Ver: 3.0.0 [ 0.431231] PCI: Using ACPI for IRQ routing [ 0.432516] NetLabel: Initializing [ 0.433007] NetLabel: domain hash size = 128 [ 0.433846] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.434078] NetLabel: unlabeled traffic allowed by default [ 0.435071] vgaarb: loaded [ 0.436138] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.437005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.441235] clocksource: Switched to clocksource kvm-clock [ 0.515518] VFS: Disk quotas dquot_6.6.0 [ 0.516422] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.517869] *** VALIDATE ramfs *** [ 0.518603] *** VALIDATE hugetlbfs *** [ 0.519440] pnp: PnP ACPI init [ 0.520905] pnp: PnP ACPI: found 6 devices [ 0.533143] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.534941] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.536175] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.537387] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.538684] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.539982] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.541623] NET: Registered protocol family 2 [ 0.543211] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.546275] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.548256] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.551816] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.553744] TCP: Hash tables configured (established 65536 bind 65536) [ 0.555330] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.557101] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.558716] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.560386] NET: Registered protocol family 1 [ 0.561865] RPC: Registered named UNIX socket transport module. [ 0.563085] RPC: Registered udp transport module. [ 0.563980] RPC: Registered tcp transport module. [ 0.564926] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.566239] NET: Registered protocol family 44 [ 0.567172] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.568388] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.569531] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.570764] PCI: CLS 0 bytes, default 64 [ 0.571632] Unpacking initramfs... [ 1.751375] debug: unmapping init [mem 0xffff8dca3cc64000-0xffff8dca3ffcffff] [ 1.753759] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.754910] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 1.756577] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.203048] Initialise system trusted keyrings [ 2.204100] Key type blacklist registered [ 2.205260] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.211727] zbud: loaded [ 2.214020] *** VALIDATE nfs *** [ 2.214822] *** VALIDATE nfs4 *** [ 2.215961] pstore: using deflate compression [ 2.217779] Platform Keyring initialized [ 2.299191] NET: Registered protocol family 38 [ 2.301089] Key type asymmetric registered [ 2.302665] Asymmetric key parser 'x509' registered [ 2.304595] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.307648] io scheduler mq-deadline registered [ 2.309500] io scheduler kyber registered [ 2.311167] io scheduler bfq registered [ 2.313040] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.316053] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.318872] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.321929] ACPI: Power Button [PWRF] [ 2.326534] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.331387] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.338621] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.366296] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.395095] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.400106] Non-volatile memory driver v1.3 [ 2.401584] Linux agpgart interface v0.103 [ 2.429577] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.432483] vda: detected capacity change from 0 to 76681216 [ 2.447450] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.450278] vdb: detected capacity change from 0 to 1073741824 [ 2.454916] libphy: Fixed MDIO Bus: probed [ 2.458849] usbcore: registered new interface driver usbserial_generic [ 2.460590] usbserial: USB Serial support registered for generic [ 2.462105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.464857] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.466028] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.467661] mousedev: PS/2 mouse device common for all mice [ 2.469805] rtc_cmos 00:05: RTC can wake from S4 [ 2.472069] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.475226] rtc_cmos 00:05: registered as rtc0 [ 2.476302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.476860] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.480418] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.481180] intel_pstate: CPU model not supported [ 2.486512] hid: raw HID events driver (C) Jiri Kosina [ 2.488601] usbcore: registered new interface driver usbhid [ 2.490645] usbhid: USB HID core driver [ 2.492181] drop_monitor: Initializing network drop monitor service [ 2.494552] Initializing XFRM netlink socket [ 2.495751] NET: Registered protocol family 10 [ 2.498081] Segment Routing with IPv6 [ 2.499292] NET: Registered protocol family 17 [ 2.500959] mpls_gso: MPLS GSO support [ 2.505058] RAS: Correctable Errors collector initialized. [ 2.506825] AVX version of gcm_enc/dec engaged. [ 2.508280] AES CTR mode by8 optimization enabled [ 2.575806] sched_clock: Marking stable (2575779088, 0)->(3172280377, -596501289) [ 2.579682] registered taskstats version 1 [ 2.581787] Loading compiled-in X.509 certificates [ 2.584022] zswap: loaded using pool lzo/zbud [ 2.607884] Key type big_key registered [ 2.618372] Key type encrypted registered [ 2.619490] ima: No TPM chip found, activating TPM-bypass! [ 2.620728] ima: Allocated hash algorithm: sha1 [ 2.622246] ima: No architecture policies found [ 2.623589] evm: Initialising EVM extended attributes: [ 2.624782] evm: security.selinux [ 2.625650] evm: security.ima [ 2.626499] evm: security.capability [ 2.627358] evm: HMAC attrs: 0x1 [ 2.628952] rtc_cmos 00:05: setting system clock to 2026-09-14 06:13:44 UTC (1789366424) [ 2.633297] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 2.635122] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 2.643070] Write protecting the kernel read-only data: 28672k [ 2.645594] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 2.647351] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 2.672215] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.680444] systemd[1]: Detected virtualization kvm. [ 2.682143] systemd[1]: Detected architecture x86-64. [ 2.683263] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.709911] systemd[1]: No hostname configured. [ 2.713511] systemd[1]: Set hostname to . [ 2.717315] random: systemd: uninitialized urandom read (16 bytes read) [ 2.720093] systemd[1]: Initializing machine ID from random generator. [ 2.765144] random: ln: uninitialized urandom read (6 bytes read) [ 2.830438] random: systemd: uninitialized urandom read (16 bytes read) [ 2.833569] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 2.839814] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 2.842908] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.369678] device-mapper: uevent: version 1.0.3 [ 3.371699] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] [ 3.961584] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.028356] virtio_net virtio0 ens2: renamed from eth0 [ 4.068327] scsi host0: ata_piix [ 4.077391] scsi host1: ata_piix [ 4.079288] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.081708] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.141649] random: crng init done [ 9.142640] random: 7 urandom warning(s) missed due to ratelimiting [ 9.241483] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.530188] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.847693] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.237482] SELinux: Disabled at runtime. [ 13.304744] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.312311] systemd[1]: Detected virtualization kvm. [ 13.314156] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.967964] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.983515] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.012858] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.027239] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.036507] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.071214] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.095362] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ 15.619130] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.236220] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.965063] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.681424] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.840438] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.218932] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 22.419562] Key type dns_resolver registered [ 22.891563] NFS: Registering the id_resolver key type [ 22.893689] Key type id_resolver registered [ 22.895394] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-client login: [ 63.565098] libcfs: loading out-of-tree module taints kernel. [ 63.677558] Key type ._llcrypt registered [ 63.679571] Key type .llcrypt registered [ 63.954084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.960593] alg: No test for adler32 (adler32-zlib) [ 65.142764] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 65.604250] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 67.247153] Key type lgssc registered [ 68.082615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.966125] Lustre: Mounted lustre-client - version 2.17.58_6_ge9bcf18 [ 131.018815] hrtimer: interrupt took 2963104 ns [ 137.566469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 154.711863] Lustre: lustre-OST0000-osc-ffff8dca85f5b000: disconnect after 23s idle [ 159.848802] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing check_logdir /tmp/testlogs/ [ 169.501145] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing yml_node [ 175.706383] Lustre: DEBUG MARKER: Client: 2.17.58.6 [ 180.761517] Lustre: DEBUG MARKER: MDS: 2.17.58.6 [ 185.214407] Lustre: DEBUG MARKER: OSS: 2.17.58.6 [ 188.719499] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 14 02:16:47 EDT 2026 [ 204.206198] Lustre: Unmounted lustre-client [ 228.899124] Key type lgssc unregistered [ 229.286868] LNet: 8407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.295263] LNetError: 8407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 229.326055] LNet: Removed LNI 192.168.201.25@tcp [ 230.057166] Key type .llcrypt unregistered [ 230.059382] Key type ._llcrypt unregistered [ 252.698567] Key type ._llcrypt registered [ 252.701282] Key type .llcrypt registered [ 252.850077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 252.873586] alg: No test for adler32 (adler32-zlib) [ 253.788966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 253.870108] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 253.897694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 258.089740] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 258.152983] LNet: 9280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.162260] LNetError: 9280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.179165] LNet: Removed LNI 192.168.201.25@tcp [ 258.780409] Key type .llcrypt unregistered [ 258.784088] Key type ._llcrypt unregistered [ 264.197822] Key type ._llcrypt registered [ 264.199270] Key type .llcrypt registered [ 265.306972] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:18:05 (1789366685) [ 265.813274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 265.827225] alg: No test for adler32 (adler32-zlib) [ 266.753385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.838484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.862624] LNet: 9694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.878583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.914709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 266.928427] LNet: 9701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 272.763593] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:18:13 (1789366693) [ 272.886469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.919162] LNet: 10278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.932732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.971902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 278.794102] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:18:19 (1789366699) [ 278.940850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 278.963789] LNet: 10860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.011597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.084996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 285.063993] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:18:25 (1789366705) [ 285.164443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.185824] LNet: 11443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.214964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.298280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 292.352834] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:18:32 (1789366712) [ 292.459516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.507883] LNet: 12025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.541774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.614286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 293.297656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 299.331199] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:18:39 (1789366719) [ 299.417433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.459321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.565597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 299.860985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 300.476322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 306.538529] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:18:47 (1789366727) [ 306.819538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.872080] LNet: 13218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.881989] LNet: 13218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 306.912548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.991352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 313.855253] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:18:54 (1789366734) [ 314.168851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.228321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.288918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 314.332471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 314.614573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 314.639172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 315.059192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 315.084488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 315.342875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 315.423624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 315.980159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 316.005541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 316.673713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 316.758066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 317.260372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 317.314451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 317.766155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 317.810367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 318.270757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 318.305992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 318.655302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 318.683896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 318.946890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 318.974344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 319.247924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 319.287193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 319.537608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 319.554816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 325.585658] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:19:06 (1789366746) [ 325.636658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.658410] LNet: 14477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.661533] LNet: 14477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 325.686277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.751642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 325.790604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 332.390491] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:19:12 (1789366752) [ 332.614618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.644560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.708248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 332.747260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 339.741252] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:19:20 (1789366760) [ 339.836921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.884266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.012181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 340.050097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 346.064846] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:19:26 (1789366766) [ 346.172840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.223929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.319295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 346.351672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 352.634385] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:19:33 (1789366773) [ 352.746751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.793861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.866081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 352.901619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 359.605774] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:19:39 (1789366779) [ 359.714836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.761929] LNet: 17413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.770420] LNet: 17413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 359.807391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.015211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 360.063463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 366.700905] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:19:47 (1789366787) [ 366.803472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.868548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.042983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 367.600181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 374.859404] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:19:55 (1789366795) [ 375.154817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.214785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.535525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.783762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 383.148924] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:20:03 (1789366803) [ 383.453167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.534632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.670943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 383.963695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 391.098723] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:20:11 (1789366811) [ 391.429361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.512598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.921290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.248563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 400.554571] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:20:20 (1789366820) [ 400.661233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.711381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.814219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.864967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.515943] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:20:27 (1789366827) [ 407.610313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.672625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.801295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 407.893818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.014696] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:20:34 (1789366834) [ 414.204522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.224791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.278838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 421.885149] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:20:41 (1789366841) [ 422.020201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.121592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.312640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 431.401690] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:20:51 (1789366851) [ 431.981547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.996988] LNet: 22858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.000821] LNet: 22858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 432.036568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.228286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 432.849241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 439.707397] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:21:00 (1789366860) [ 439.844673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.898697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.980620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 440.027392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 447.138455] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:21:07 (1789366867) [ 447.357694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.382650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.427404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 447.450353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 454.445761] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:21:14 (1789366874) [ 454.649306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.783768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.868767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 454.906449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 462.534452] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:21:23 (1789366883) [ 462.617144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.681067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.795249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 462.828801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 469.500944] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:21:30 (1789366890) [ 469.633452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.710357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.810721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 469.887727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 469.902974] LNetError: 25807:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 469.949966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 469.980559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 476.199295] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:21:36 (1789366896) [ 476.401441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.449358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.529489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DnZZDSwZaa [ 476.645334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DnZZDSwZaa [ 476.707597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DnZZDSwZaa [ 483.411394] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:21:43 (1789366903) [ 483.503278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.594224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.734097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 483.816837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 483.891457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 489.972619] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:21:50 (1789366910) [ 490.079197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.125757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.243459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 490.277958] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 491.737445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 493.037505] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 495.056917] Key type lgssc registered [ 496.273866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 500.630193] LNetError: 27567:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.25@tcp99 has no available nets [ 503.180630] Key type lgssc unregistered [ 503.508241] LNetError: 28673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.517342] LNet: Removed LNI 192.168.201.25@tcp [ 504.017735] Key type .llcrypt unregistered [ 504.021161] Key type ._llcrypt unregistered [ 510.460196] Key type ._llcrypt registered [ 510.464568] Key type .llcrypt registered [ 511.830184] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:22:12 (1789366932) [ 512.813373] Key type .llcrypt unregistered [ 512.815826] Key type ._llcrypt unregistered [ 519.589862] Key type ._llcrypt registered [ 519.591984] Key type .llcrypt registered [ 519.866375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 519.888685] alg: No test for adler32 (adler32-zlib) [ 520.784335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.848976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 520.869721] LNet: 29583:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 520.879116] LNet: 29583:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 520.888094] LNet: 29583:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 520.904249] LNet: Message forwarding will be enabled in 2s [ 523.999159] LNet: Message forwarding enabled [ 524.390862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 524.431634] LNet: Message forwarding will stop in 2s to 6s [ 528.097723] LNet: Message forwarding disabled [ 529.032374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 529.052211] LNet: 29682:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 529.057634] LNet: 29682:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 529.066292] LNet: 29682:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 529.082233] LNet: Message forwarding will be enabled in 2s [ 529.276948] LNet: 29695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.872398] Key type .llcrypt unregistered [ 529.876850] Key type ._llcrypt unregistered [ 543.184940] Key type ._llcrypt registered [ 543.193681] Key type .llcrypt registered [ 544.572586] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:22:44 (1789366964) [ 544.959909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.972360] alg: No test for adler32 (adler32-zlib) [ 545.892831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.991713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 552.372768] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:22:52 (1789366972) [ 552.509508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.544219] LNet: 31088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.562508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.647876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 552.685685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 552.715689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 552.741812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 552.765680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 552.789538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 552.826541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 552.857798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 552.918980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 552.955647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 552.995965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 553.057568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 553.083685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 553.117654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 553.154637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 553.184734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 553.219705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 553.278913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 553.324701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 553.358996] LNetError: 31138:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 560.780948] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:23:00 (1789366980) [ 561.057242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.081835] LNet: 31709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.102592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.245060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.713717] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:23:09 (1789366989) [ 571.886661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg125-server [ 573.744688] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:23:14 (1789366994) [ 574.284398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.336260] LNet: 32640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.356769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.410931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 574.455669] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 575.534579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[26-30]@tcp [ 575.551620] LNet: 32774:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 575.600744] LNetError: 32644:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 575.612477] LNetError: 32644:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.28@tcp has gone from up to down [ 578.727339] LNetError: 32775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.30@tcp has gone from up to down [ 578.739934] LNetError: 32775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 582.721030] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:23:23 (1789367003) [ 584.723670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg125-server [ 586.638169] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:23:27 (1789367007) [ 586.774281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.829625] LNet: 33702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.848169] LNetError: 33702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.878499] LNet: Removed LNI 192.168.201.25@tcp [ 586.919294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.020956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.047741] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 587.718882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[26-31/2]@tcp [ 587.755192] LNet: 33827:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 587.762197] LNet: 33827:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 587.803569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[26-31/2]@tcp [ 587.804049] LNetError: 33712:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 587.852547] LNetError: 33712:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 587.862561] LNetError: 33712:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.28@tcp has gone from up to down [ 594.961379] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:23:35 (1789367015) [ 595.064109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.101032] LNet: 34411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.108210] LNetError: 34411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.135675] LNet: Removed LNI 192.168.201.25@tcp [ 595.160274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.302152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 595.359663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.392475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.481229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.526834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.569791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.652345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 595.736956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.821817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.945162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 596.037641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.080883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.182853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 596.273496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.323614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.384406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 596.445289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.472429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.513891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 596.604864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.674651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.788859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 596.879453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.938570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.998701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 597.089683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.130597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.193257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.277399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.299023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.344778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 597.411346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.448227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.508559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.959200] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:23:43 (1789367023) [ 604.944778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg125-server [ 606.728359] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:23:47 (1789367027) [ 608.457409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg125-server [ 609.866916] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:23:50 (1789367030) [ 610.084486] LNet: 35819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.091326] LNet: 35819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 610.584209] Key type .llcrypt unregistered [ 610.586947] Key type ._llcrypt unregistered [ 615.472311] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 615.729820] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 615.734166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 615.749341] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 618.191151] Key type ._llcrypt registered [ 618.195397] Key type .llcrypt registered [ 618.436139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.454714] alg: No test for adler32 (adler32-zlib) [ 619.379202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.424962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.441715] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 620.096780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 620.117504] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 622.180510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 622.225822] LNet: Removed LNI 10.1.2.3@tcp [ 622.398599] LNet: 36535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.404443] LNetError: 36535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.420611] LNet: Removed LNI 192.168.201.25@tcp [ 622.913616] Key type .llcrypt unregistered [ 622.915219] Key type ._llcrypt unregistered [ 633.619313] Key type ._llcrypt registered [ 633.621302] Key type .llcrypt registered [ 634.783688] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:24:15 (1789367055) [ 635.247383] Key type .llcrypt unregistered [ 635.253925] Key type ._llcrypt unregistered [ 640.141724] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.415465] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.424109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.434509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 642.926747] Key type ._llcrypt registered [ 642.932711] Key type .llcrypt registered [ 643.135192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.151414] alg: No test for adler32 (adler32-zlib) [ 644.073969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.132023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.183804] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 644.874685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.901239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.052628] LNet: 38038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.056970] LNetError: 38038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 647.061570] LNet: Removed LNI 10.1.2.3@tcp [ 647.073858] LNet: Removed LNI 192.168.201.25@tcp [ 647.619528] Key type .llcrypt unregistered [ 647.621203] Key type ._llcrypt unregistered [ 657.389508] Key type ._llcrypt registered [ 657.391728] Key type .llcrypt registered [ 658.496581] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:24:39 (1789367079) [ 658.994559] Key type .llcrypt unregistered [ 658.999290] Key type ._llcrypt unregistered [ 664.887234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 665.148738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 665.154094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 665.162590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 667.731765] Key type ._llcrypt registered [ 667.733950] Key type .llcrypt registered [ 667.938830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 667.952341] alg: No test for adler32 (adler32-zlib) [ 668.861768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.923952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 668.943426] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 670.883991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 670.899939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 672.714738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 672.732940] LNet: Removed LNI 10.1.2.3@tcp [ 672.742727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 672.763804] LNet: Removed LNI 10.1.2.31@tcp [ 672.776931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 672.791356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 673.320877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 673.338929] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 674.226733] LNet: 39952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.231814] LNetError: 39952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 674.236978] LNet: Removed LNI 10.1.2.31@tcp [ 674.243763] LNet: Removed LNI 10.1.2.3@tcp [ 674.833653] Key type .llcrypt unregistered [ 674.837059] Key type ._llcrypt unregistered [ 686.807667] Key type ._llcrypt registered [ 686.810223] Key type .llcrypt registered [ 688.329538] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:25:08 (1789367108) [ 688.850841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.869512] alg: No test for adler32 (adler32-zlib) [ 689.793787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.863678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.915228] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 690.877461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.898293] LNet: 40902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.903988] LNetError: 40902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.918940] LNet: Removed LNI 192.168.201.25@tcp [ 690.936355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.989923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.25@tcp [ 691.019956] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 691.931481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.25@tcp [ 691.959341] LNet: Removed LNI 192.168.201.25@tcp [ 692.023276] LNet: 41037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.622683] Key type .llcrypt unregistered [ 692.624809] Key type ._llcrypt unregistered [ 703.706598] Key type ._llcrypt registered [ 703.713254] Key type .llcrypt registered [ 704.927238] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:25:25 (1789367125) [ 706.875371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg125-server [ 708.999335] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:25:29 (1789367129) [ 709.485894] Key type .llcrypt unregistered [ 709.487755] Key type ._llcrypt unregistered [ 715.850378] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 716.035831] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.040725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.047532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 718.659375] Key type ._llcrypt registered [ 718.662431] Key type .llcrypt registered [ 718.826686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 718.849815] alg: No test for adler32 (adler32-zlib) [ 719.754465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.802107] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 720.559228] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 722.086568] LNet: 42869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.093637] LNetError: 42869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.096974] LNet: Removed LNI 192.168.201.25@tcp1 [ 722.106317] LNet: Removed LNI 10.1.2.3@tcp2 [ 722.594530] Key type .llcrypt unregistered [ 722.596637] Key type ._llcrypt unregistered [ 732.655698] Key type ._llcrypt registered [ 732.658233] Key type .llcrypt registered [ 733.937020] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:25:54 (1789367154) [ 734.269519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.292308] alg: No test for adler32 (adler32-zlib) [ 735.225754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.284780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.317792] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 736.181685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.212882] LNet: 43834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.222072] LNetError: 43834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.237771] LNet: Removed LNI 192.168.201.25@tcp [ 736.259714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.346774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 736.377321] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 741.759078] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:26:02 (1789367162) [ 741.863088] LNet: 44421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.871654] LNetError: 44421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.893385] LNet: Removed LNI 192.168.201.25@tcp [ 742.451762] Key type .llcrypt unregistered [ 742.455771] Key type ._llcrypt unregistered [ 747.628837] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.828075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.830519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.838301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.321760] Key type ._llcrypt registered [ 750.326412] Key type .llcrypt registered [ 750.531706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.540489] alg: No test for adler32 (adler32-zlib) [ 751.438882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.480475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.502476] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 752.166381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 752.184944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 753.827874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.843456] LNet: 45166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.848268] LNetError: 45166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 753.857371] LNet: Removed LNI 10.1.2.3@tcp [ 753.868885] LNet: Removed LNI 192.168.201.25@tcp [ 753.896262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.979612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 754.008095] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 754.012920] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 754.595577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.618938] LNet: 45193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.625689] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 754.638305] LNetError: 45193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 754.647265] LNet: Removed LNI 10.1.2.3@tcp [ 754.659576] LNet: Removed LNI 192.168.201.25@tcp [ 754.678386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.733505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 754.756262] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 754.766446] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 755.221294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 755.239672] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 756.272406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.299297] LNet: 45355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.303990] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 756.309602] LNetError: 45355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 756.317222] LNet: Removed LNI 10.1.2.3@tcp [ 756.320116] LNet: Removed LNI 192.168.201.25@tcp [ 756.339466] LNet: Removed LNI 192.168.201.25@tcp2 [ 756.408434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.499607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 756.537459] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 756.547492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.557193] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 762.999194] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:26:23 (1789367183) [ 763.108111] LNet: 45947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.123405] LNetError: 45947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 763.128898] LNetError: 45947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 763.137774] LNet: Removed LNI 10.1.2.3@tcp [ 763.142813] LNet: Removed LNI 192.168.201.25@tcp [ 763.178736] LNet: Removed LNI 192.168.201.25@tcp2 [ 763.709640] Key type .llcrypt unregistered [ 763.711043] Key type ._llcrypt unregistered [ 768.842279] Key type ._llcrypt registered [ 768.844734] Key type .llcrypt registered [ 769.123213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.141904] alg: No test for adler32 (adler32-zlib) [ 770.043299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.114593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.162120] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 771.119664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.137691] LNet: 46345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.143593] LNetError: 46345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.168031] LNet: Removed LNI 192.168.201.25@tcp [ 771.188932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.241541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 771.272510] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 771.781653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.800598] LNet: 46375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.810709] LNetError: 46375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.839779] LNet: Removed LNI 192.168.201.25@tcp [ 771.893128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.986234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 772.015542] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 772.465342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.518584] LNet: Removed LNI 192.168.201.25@tcp [ 772.543526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.631243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 772.664347] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 773.126316] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 773.290556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 773.302582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.308359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.673478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.689293] LNet: 46483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.692809] LNet: 46483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 775.698540] LNetError: 46483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.704174] LNetError: 46483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 775.717919] LNet: Removed LNI 192.168.201.25@tcp [ 775.751758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.834769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 775.869547] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 781.044768] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:26:41 (1789367201) [ 781.202704] LNet: 47077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.216308] LNetError: 47077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.236966] LNet: Removed LNI 192.168.201.25@tcp [ 781.779127] Key type .llcrypt unregistered [ 781.780746] Key type ._llcrypt unregistered [ 786.969625] Key type ._llcrypt registered [ 786.973259] Key type .llcrypt registered [ 787.140355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.149712] alg: No test for adler32 (adler32-zlib) [ 788.106500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.162849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.206935] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 788.964794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.982790] LNet: 47477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.989178] LNetError: 47477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.998276] LNet: Removed LNI 192.168.201.25@tcp [ 789.019712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.090959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 789.115797] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 795.086888] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:26:55 (1789367215) [ 795.325270] LNet: 48069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.333101] LNetError: 48069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.357851] LNet: Removed LNI 192.168.201.25@tcp [ 795.833562] Key type .llcrypt unregistered [ 795.837078] Key type ._llcrypt unregistered [ 800.636437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.777111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.791751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.801844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.268625] Key type ._llcrypt registered [ 803.272680] Key type .llcrypt registered [ 803.461613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.477400] alg: No test for adler32 (adler32-zlib) [ 804.369203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.415646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.449737] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 805.061990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.076342] LNet: 48521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.081413] LNetError: 48521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.108328] LNet: Removed LNI 192.168.201.25@tcp [ 805.129996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.210624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 805.229392] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 811.173917] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:27:11 (1789367231) [ 811.269502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.292898] LNet: 49116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.325623] LNetError: 49116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.341264] LNet: Removed LNI 192.168.201.25@tcp [ 811.354389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.437073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 817.112919] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:27:17 (1789367237) [ 817.318173] LNet: 49700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.964088] Key type .llcrypt unregistered [ 817.970500] Key type ._llcrypt unregistered [ 823.372373] Key type ._llcrypt registered [ 823.374524] Key type .llcrypt registered [ 823.518871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.537460] alg: No test for adler32 (adler32-zlib) [ 824.428314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.469819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.486700] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 825.161167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.176584] LNet: 50095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.182856] LNetError: 50095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.193202] LNet: Removed LNI 192.168.201.25@tcp [ 825.211420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.266707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 825.284621] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 830.305452] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:27:31 (1789367251) [ 830.391501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.414883] LNet: 50682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.427965] LNetError: 50682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.448981] LNet: Removed LNI 192.168.201.25@tcp [ 830.508347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.598736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 835.451527] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:27:36 (1789367256) [ 836.694856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 837.997572] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:27:38 (1789367258) [ 839.383926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 841.150805] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:27:41 (1789367261) [ 842.332862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 843.786345] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:27:44 (1789367264) [ 845.002791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 846.746742] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:27:47 (1789367267) [ 847.766287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 849.069593] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:27:49 (1789367269) [ 849.201993] LNet: 52707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.689565] Key type .llcrypt unregistered [ 849.691916] Key type ._llcrypt unregistered [ 854.236665] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 854.386130] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 854.389781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 854.394621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 856.827616] Key type ._llcrypt registered [ 856.829268] Key type .llcrypt registered [ 856.956455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.963732] alg: No test for adler32 (adler32-zlib) [ 857.827787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.883289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 857.901483] LNet: Added LNI 192.168.201.25@tcp [8/512/0/180] [ 857.905016] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 857.908386] LNet: 53020:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 857.993595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.010587] LNet: 53039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.018710] LNetError: 53039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.025942] LNet: Removed LNI 10.1.2.3@tcp [ 858.031288] LNetError: 53039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.042888] LNet: Removed LNI 192.168.201.25@tcp [ 858.065492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.113610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.133773] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 858.188336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.208365] LNet: 53063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.215335] LNetError: 53063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.232588] LNet: Removed LNI 192.168.201.25@tcp [ 858.249856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.305802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.333107] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 858.342888] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.344449] LNet: 53071:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 858.410560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.427149] LNet: 53089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.431644] LNetError: 53089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.437783] LNet: Removed LNI 10.1.2.3@tcp [ 858.439308] LNetError: 53089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.450668] LNet: Removed LNI 192.168.201.25@tcp [ 858.464153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.521979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.538892] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 858.543648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.546240] LNet: 53097:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 858.687940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.702462] LNet: 53123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.707510] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.714148] LNet: Removed LNI 10.1.2.3@tcp [ 858.719871] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.735539] LNet: Removed LNI 192.168.201.25@tcp [ 858.757718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.810625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 858.826663] LNet: Added LNI 192.168.201.25@tcp [8/256/0/182] [ 858.831120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 858.907239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.919958] LNet: 53151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.923840] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.927144] LNet: Removed LNI 10.1.2.3@tcp [ 858.929922] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.938654] LNet: Removed LNI 192.168.201.25@tcp [ 858.952903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.006899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 859.034589] LNet: Added LNI 192.168.201.25@tcp [8/256/0/182] [ 859.038742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 859.044967] LNet: 53159:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 859.108609] LNet: Removed LNI 10.1.2.3@tcp [ 859.119730] LNet: Removed LNI 192.168.201.25@tcp [ 859.586463] Key type .llcrypt unregistered [ 859.588905] Key type ._llcrypt unregistered [ 868.577572] Key type ._llcrypt registered [ 868.579819] Key type .llcrypt registered [ 869.589357] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:28:10 (1789367290) [ 870.756395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 872.463110] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:28:12 (1789367292) [ 872.830939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.845717] alg: No test for adler32 (adler32-zlib) [ 873.804400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.881551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.919689] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 874.599941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.25 [ 874.615883] LNet: 54420:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 874.622448] LNet: Removed LNI 192.168.201.25@tcp [ 879.354176] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:28:20 (1789367300) [ 879.435217] LNet: 54981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.441409] LNetError: 54981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.453188] LNet: Removed LNI 192.168.201.25@tcp [ 879.952702] Key type .llcrypt unregistered [ 879.956865] Key type ._llcrypt unregistered [ 884.902808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.172378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.848293] Key type ._llcrypt registered [ 887.851361] Key type .llcrypt registered [ 888.017275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.027762] alg: No test for adler32 (adler32-zlib) [ 889.112047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.053413] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:28:34 (1789367314) [ 894.164974] LNet: 55897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.170665] LNetError: 55897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.184189] LNet: Removed LNI 10.1.2.3@tcp [ 894.874786] Key type .llcrypt unregistered [ 894.879285] Key type ._llcrypt unregistered [ 899.376470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.600528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.615358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.619252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.154713] Key type ._llcrypt registered [ 902.157427] Key type .llcrypt registered [ 902.268873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.277616] alg: No test for adler32 (adler32-zlib) [ 903.210561] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.077730] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:28:48 (1789367328) [ 909.295521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 910.433218] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:28:51 (1789367331) [ 910.894885] LNet: 57082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.898527] LNetError: 57082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.915931] LNet: Removed LNI 10.1.2.3@tcp [ 911.381551] Key type .llcrypt unregistered [ 911.384038] Key type ._llcrypt unregistered [ 915.922613] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.187436] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.193170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.201854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.774551] Key type ._llcrypt registered [ 918.777166] Key type .llcrypt registered [ 918.913938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.925699] alg: No test for adler32 (adler32-zlib) [ 919.786313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.825509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.819737] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:29:06 (1789367346) [ 926.157484] LNet: 58308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.162308] LNetError: 58308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.174215] LNet: Removed LNI 10.1.2.3@tcp [ 926.686814] Key type .llcrypt unregistered [ 926.688948] Key type ._llcrypt unregistered [ 932.034988] Key type ._llcrypt registered [ 932.037890] Key type .llcrypt registered [ 932.196686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.208764] alg: No test for adler32 (adler32-zlib) [ 933.091126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 933.124788] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 939.347293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 940.004718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 940.025845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 940.076094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 940.095663] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 940.298696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 940.319321] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 940.343918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 940.383253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 940.415360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 940.447852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 940.482247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 940.526598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 940.569895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 940.616862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 940.644426] LNet: 58745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 940.674181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 940.716462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 940.746893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 940.788113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 940.832503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 940.871686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 940.907667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 941.320444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 941.353762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 941.383625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 941.398763] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 941.607734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.638915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 941.666316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.691409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.716861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 941.762503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.821065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.850802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.867918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.878343] LNet: 58856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 941.891961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.910694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.930926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.956481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.989466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 942.013846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 942.044924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 942.493263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 942.518978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 942.543782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 942.558477] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 942.728913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.747989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.767192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.783358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.803499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 942.826300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.850876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.880101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.915324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.937964] LNet: 58968:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 942.962670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.007914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.043729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.077736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.104856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.129251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.155125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 943.625304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 943.672213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 943.708144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 943.905269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.930859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 943.959386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 943.988788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.011467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 944.034832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.060297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.084506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.111038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.145994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.170664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.192949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.222372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.262483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.288738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.324882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 944.689512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 944.739289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 944.764231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 944.783603] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 944.786351] LNet: Skipped 1 previous similar message [ 944.947329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.980485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 945.004423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.031411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.052384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 945.081869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.113469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.142899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.166922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.182441] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 945.188576] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 945.204640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.233705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.259666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.288117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.309552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.337263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.364242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 945.874788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 945.911272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 945.938316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 946.171956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.202659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 946.249256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 946.297201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 946.326436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 946.366288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 946.414879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 946.450487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 946.495971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 946.522909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 946.567981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 946.599530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 946.625290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 946.651714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 946.677334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 946.705322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 946.917855] LNet: 59342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.922588] LNetError: 59342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.933879] LNet: Removed LNI 192.168.201.25@tcp [ 947.252559] Key type .llcrypt unregistered [ 947.254314] Key type ._llcrypt unregistered [ 960.755244] Key type ._llcrypt registered [ 960.756707] Key type .llcrypt registered [ 962.014348] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:29:42 (1789367382) [ 962.712438] Key type .llcrypt unregistered [ 962.716131] Key type ._llcrypt unregistered [ 967.662486] Key type ._llcrypt registered [ 967.664220] Key type .llcrypt registered [ 967.802394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.811913] alg: No test for adler32 (adler32-zlib) [ 968.696880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 968.726195] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 974.581374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 982.077254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.127157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 982.160936] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 983.271371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 983.301343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 1 [ 983.333549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 983.369204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 1 [ 983.398839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_interrupt -r 1 [ 983.434641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 1 [ 983.473829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_interrupt -r 1 [ 983.530793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 1 [ 983.579490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 983.598240] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 983.608781] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 983.803627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.835638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 983.866659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.903358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.929781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 983.960971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.989570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 984.014939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 984.040691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 984.052673] LNet: 60806:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 984.069352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 984.101770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 984.129899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 984.167673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 984.194577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 984.216964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 984.247856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 984.670531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 984.708920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 1 [ 984.748316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 984.779560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 1 [ 984.817661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_dropped -r 1 [ 984.864020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 1 [ 984.927687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_dropped -r 1 [ 985.021982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 1 [ 985.075315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 985.095499] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 985.103930] LNet: Skipped 2 previous similar messages [ 985.304716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 985.328874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 985.357047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.377984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.416979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 985.444665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.471161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.504725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.526760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.541819] LNet: 60930:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.555843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.572282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.592841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.619989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.652509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.675748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.713632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 986.112141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 986.139666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 1 [ 986.165909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 986.197938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 1 [ 986.223850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_aborted -r 1 [ 986.261885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 1 [ 986.291949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_aborted -r 1 [ 986.326862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 1 [ 986.363179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 986.380488] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 986.384764] LNet: Skipped 2 previous similar messages [ 986.552737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.574581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 986.594763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.621251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.646689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 986.674271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.699676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.734426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.761093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 986.773487] LNet: 61053:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.787883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 986.810914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 986.843871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 986.875620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 986.903632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 986.929807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 986.952782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 987.435767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 987.463829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 1 [ 987.489501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 987.515730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 1 [ 987.538482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_no_route -r 1 [ 987.564899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 1 [ 987.594927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_no_route -r 1 [ 987.636587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 1 [ 987.683693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 987.876964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.900491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 987.934106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 987.964659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 987.996263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 988.030306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.074943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.098829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.128657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.156238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.178496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.200793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.223324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.245683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.268468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.297782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 988.671881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 988.700412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 1 [ 988.736534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 988.773914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 1 [ 988.811659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_timeout -r 1 [ 988.840914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 1 [ 988.873989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_timeout -r 1 [ 988.907646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 1 [ 988.941806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 988.966820] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 988.969968] LNet: Skipped 5 previous similar messages [ 989.173173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.234689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 989.277348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.322843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.350372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 989.376642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.397724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.421872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.443693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.456354] LNet: 61300:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.463429] LNet: 61300:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 989.478279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.505883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.546297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.572470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.601450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.626230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.650585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 990.002114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 990.026205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_error -r 1 [ 990.054859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 990.081924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_error -r 1 [ 990.109794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_error -r 1 [ 990.133747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_error -r 1 [ 990.160993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_error -r 1 [ 990.191751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_error -r 1 [ 990.223552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 990.432090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.457499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.481942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.515137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.543743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 990.567291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.589285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.614738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.641952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.668560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.728043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.766946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.808058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.828063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.853357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.888724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 995.232911] LNet: 61499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.235995] LNetError: 61499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.241849] LNet: Removed LNI 192.168.201.25@tcp [ 995.253572] LNet: Removed LNI 192.168.201.25@tcp1 [ 995.534754] Key type .llcrypt unregistered [ 995.536728] Key type ._llcrypt unregistered [ 1007.963306] Key type ._llcrypt registered [ 1007.964444] Key type .llcrypt registered [ 1008.786732] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:30:29 (1789367429) [ 1009.402538] Key type .llcrypt unregistered [ 1009.404085] Key type ._llcrypt unregistered [ 1014.022975] Key type ._llcrypt registered [ 1014.025123] Key type .llcrypt registered [ 1014.226599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.245176] alg: No test for adler32 (adler32-zlib) [ 1015.198651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1015.262654] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1022.481462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1022.999660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1023.022611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1023.047244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1023.059191] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1023.199479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.222762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.243805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.269960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.312098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1023.333941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.368872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.398271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.436764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.448238] LNet: 62745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.460282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.488384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.513649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.539862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.577257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.610457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.640338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1024.123453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1024.154282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1024.186496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1024.198454] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1024.393583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.431425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.458565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.491379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.517558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1024.552325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.598648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.635001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.673364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.687225] LNet: 62856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.717559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.762328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.797406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.828218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.859171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.890805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.925230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1025.433696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1025.474463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1025.504750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1025.518460] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1025.708810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.734306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.754325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.779875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.809491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1025.840953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.872322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.905694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.933423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.949247] LNet: 62967:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.959075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.983662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.010170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.036743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.065183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.107251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.147157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.424992] LNet: 63008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.433907] LNetError: 63008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.449933] LNet: Removed LNI 192.168.201.25@tcp [ 1026.785112] Key type .llcrypt unregistered [ 1026.789363] Key type ._llcrypt unregistered [ 1040.147858] Key type ._llcrypt registered [ 1040.150168] Key type .llcrypt registered [ 1041.412925] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:31:01 (1789367461) [ 1042.227537] Key type .llcrypt unregistered [ 1042.229436] Key type ._llcrypt unregistered [ 1047.637773] Key type ._llcrypt registered [ 1047.640166] Key type .llcrypt registered [ 1047.878308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1047.900875] alg: No test for adler32 (adler32-zlib) [ 1048.846941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1048.881435] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1055.107653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1062.501462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.549600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1062.572874] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1063.650354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1063.676764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 1 [ 1063.727364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1063.767302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 1 [ 1063.798391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_dropped -r 1 [ 1063.824433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 1 [ 1063.841457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_dropped -r 1 [ 1063.858155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 1 [ 1063.881168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1063.895553] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1063.900043] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1064.150950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.188937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.222596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.254459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.284964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1064.327902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.358604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.387155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.415880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.432923] LNet: 64471:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.448435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.480697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.517203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.546711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.593529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.616728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.640307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.677763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.056301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1065.078402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_error -r 1 [ 1065.105988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1065.132760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_error -r 1 [ 1065.158954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_error -r 1 [ 1065.189582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_error -r 1 [ 1065.216770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_error -r 1 [ 1065.243678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_error -r 1 [ 1065.263882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1065.273986] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1065.277325] LNet: Skipped 2 previous similar messages [ 1065.470660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.518259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.566723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.592292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.616059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1065.641639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.691903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.715906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.738764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.748420] LNet: 64596:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.758637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.774440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.794624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.818499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.840739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.863760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.899800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.927644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.378403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1066.404832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_timeout -r 1 [ 1066.435392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1066.461785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_timeout -r 1 [ 1066.489498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_timeout -r 1 [ 1066.517106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_timeout -r 1 [ 1066.540283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_timeout -r 1 [ 1066.558913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_timeout -r 1 [ 1066.584312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.125@tcp [ 1066.598193] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1066.750775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.768117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.795244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.828574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.853251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1066.876834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.905513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.930904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.960053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.978371] LNet: 64721:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.997927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.029759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.076598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.137631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.175626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.192145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.209251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1067.232171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.598737] LNet: 64799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.605485] LNetError: 64799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.613781] LNet: Removed LNI 192.168.201.25@tcp [ 1071.625399] LNet: Removed LNI 192.168.201.25@tcp1 [ 1072.031712] Key type .llcrypt unregistered [ 1072.033962] Key type ._llcrypt unregistered [ 1086.497818] Key type ._llcrypt registered [ 1086.499905] Key type .llcrypt registered [ 1087.590518] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:31:48 (1789367508) [ 1088.847068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1090.540803] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:31:51 (1789367511) [ 1091.355659] Key type .llcrypt unregistered [ 1091.357105] Key type ._llcrypt unregistered [ 1096.391181] Key type ._llcrypt registered [ 1096.393250] Key type .llcrypt registered [ 1096.534528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.548202] alg: No test for adler32 (adler32-zlib) [ 1097.422789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1097.447259] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1103.665475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1104.474826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1104.517962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1104.557230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1104.575828] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1104.607128] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1104.838706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.873320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.916887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.948365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.984252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1105.020165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.063770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.098733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.137437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.174708] LNet: 66352:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.204308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.240268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.287636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.316818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.365150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.409180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.718573] LNet: 66391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.726573] LNetError: 66391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.760925] LNet: Removed LNI 192.168.201.25@tcp [ 1106.267515] Key type .llcrypt unregistered [ 1106.269728] Key type ._llcrypt unregistered [ 1120.994954] Key type ._llcrypt registered [ 1120.998472] Key type .llcrypt registered [ 1121.195571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.220173] alg: No test for adler32 (adler32-zlib) [ 1122.103763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1122.133952] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1128.728961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1136.306936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.340593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1136.358942] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1137.472043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1137.508736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e network_timeout -r 1 [ 1137.555614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1137.621502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e network_timeout -r 1 [ 1137.697753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e network_timeout -r 1 [ 1137.778162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e network_timeout -r 1 [ 1137.836901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e network_timeout -r 1 [ 1137.893316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e network_timeout -r 1 [ 1137.940655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1137.964630] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1138.175580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.214609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.246931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.274598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.300630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1138.337046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.372207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.407567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.452574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.471603] LNet: 67266:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.506195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.527344] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1138.531810] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1138.532451] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1138.553519] LNet: Skipped 1 previous similar message [ 1138.565320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.598311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.629624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.671225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.706286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.051581] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1143.432948] LNet: 67340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.439706] LNetError: 67340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.446289] LNet: Removed LNI 192.168.201.25@tcp [ 1143.462472] LNet: Removed LNI 192.168.201.25@tcp1 [ 1143.832304] Key type .llcrypt unregistered [ 1143.834446] Key type ._llcrypt unregistered [ 1159.409197] Key type ._llcrypt registered [ 1159.411500] Key type .llcrypt registered [ 1160.936079] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:33:01 (1789367581) [ 1161.292403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1161.312493] alg: No test for adler32 (adler32-zlib) [ 1162.203429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.247843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1162.265712] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1162.765659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1162.785792] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1163.450225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1163.534767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1163.574835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1163.604676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1163.651216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1163.685527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1163.723892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 1163.748357] LNetError: 68438:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1164.255667] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1164.256161] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1166.304937] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1167.276221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1170.399520] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1170.830996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1170.887971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.959201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.002309] LNet: 68545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.008864] LNetError: 68545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.017818] LNet: Removed LNI 192.168.201.25@tcp [ 1171.032921] LNet: Removed LNI 192.168.201.25@tcp1 [ 1171.064273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.120630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1171.150988] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1171.817250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1171.834469] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1172.465609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1172.501830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.554829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1172.595137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1172.644923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1172.696471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1172.739042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1172.781310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 1172.798861] LNetError: 68817:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1173.152896] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1176.245682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1179.295591] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1179.300443] LNet: Skipped 1 previous similar message [ 1184.037133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1184.072295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1184.104870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.446066] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:33:29 (1789367609) [ 1188.567938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1188.588168] LNet: 69531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.598700] LNetError: 69531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.608605] LNetError: 69531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1188.614574] LNet: Removed LNI 192.168.201.25@tcp [ 1188.627730] LNet: Removed LNI 192.168.201.25@tcp1 [ 1188.643981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.669963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1188.692419] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1189.104688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.123369] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1189.653287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1189.714856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1189.741772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1189.773187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1189.805612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1189.834587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1189.858282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 0 [ 1189.936290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1190.687272] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.25@tcp [ 1190.687324] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1190.697663] LNet: Skipped 1 previous similar message [ 1205.114456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1205.263456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1205.310785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.349413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 500 [ 1205.445389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1212.065099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1212.112200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1212.128566] LNet: 70024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.134190] LNetError: 70024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.138585] LNetError: 70024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1212.145092] LNet: Removed LNI 192.168.201.25@tcp [ 1212.162685] LNet: Removed LNI 192.168.201.25@tcp1 [ 1212.178093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.224531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.255391] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1212.904467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.926342] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1213.553943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1213.590804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.653457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1213.680179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1213.709420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1213.737735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1213.763589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1213.799842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.25@tcp --health 0 [ 1214.239474] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1214.245634] LNet: Skipped 3 previous similar messages [ 1215.103309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1225.149539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1225.185607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1225.219850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.471651] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:34:11 (1789367651) [ 1230.837880] LNet: 71070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.846838] LNetError: 71070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.851785] LNetError: 71070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1230.855950] LNet: Removed LNI 192.168.201.25@tcp [ 1230.868619] LNet: Removed LNI 192.168.201.25@tcp1 [ 1231.327818] Key type .llcrypt unregistered [ 1231.330254] Key type ._llcrypt unregistered [ 1236.012557] Key type ._llcrypt registered [ 1236.015831] Key type .llcrypt registered [ 1236.197561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1236.210803] alg: No test for adler32 (adler32-zlib) [ 1237.083694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1237.102723] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1243.621161] Lustre: DEBUG MARKER: Initial discovery [ 1243.689179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1245.285244] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1245.449882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1245.482830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1245.503106] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1245.519700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.544713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.125@tcp -r 1 -m GET -l 3 [ 1245.578547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.125@tcp -r 1 -m GET -e local_error [ 1245.610809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.125@tcp -r 1 -m PUT -l 6 [ 1246.308287] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1246.505839] Lustre: DEBUG MARKER: Force oleg125-client.virtnet to discover 192.168.201.125@tcp (in background) [ 1246.622568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.125@tcp [ 1247.664293] Lustre: DEBUG MARKER: Force oleg125-server to discover 192.168.201.25@tcp [ 1249.286530] Lustre: DEBUG MARKER: Wait for 71827 [ 1249.695731] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1250.740281] Lustre: DEBUG MARKER: Finished wait on 71827 [ 1250.778958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1250.818883] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1250.824367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1251.043456] LNet: 72279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.048435] LNetError: 72279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.058448] LNet: Removed LNI 192.168.201.25@tcp [ 1251.338767] Key type .llcrypt unregistered [ 1251.341560] Key type ._llcrypt unregistered [ 1263.276692] Key type ._llcrypt registered [ 1263.282837] Key type .llcrypt registered [ 1264.404628] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:34:45 (1789367685) [ 1264.866476] Key type .llcrypt unregistered [ 1264.871891] Key type ._llcrypt unregistered [ 1269.017266] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1269.139677] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1269.143931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1269.147705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1271.530303] Key type ._llcrypt registered [ 1271.531650] Key type .llcrypt registered [ 1271.656697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1271.669676] alg: No test for adler32 (adler32-zlib) [ 1272.558188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.598450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1272.616162] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1273.204474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1273.221980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1279.184892] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:35:00 (1789367700) [ 1279.293420] LNet: 74347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.297858] LNetError: 74347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1279.301892] LNet: Removed LNI 10.1.2.3@tcp [ 1279.315340] LNet: Removed LNI 192.168.201.25@tcp [ 1279.762644] Key type .llcrypt unregistered [ 1279.764570] Key type ._llcrypt unregistered [ 1283.802211] Key type ._llcrypt registered [ 1283.805119] Key type .llcrypt registered [ 1283.954185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.965530] alg: No test for adler32 (adler32-zlib) [ 1284.818570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1284.847842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1284.867917] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1285.416567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1285.437433] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1286.097704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.25@tcp [ 1287.138677] LNetError: Unexpected error -22 connecting to 192.168.201.25@tcp at host 192.168.201.25:988 [ 1290.561900] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:35:11 (1789367711) [ 1290.642832] LNet: 75468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.648439] LNetError: 75468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1290.653797] LNet: Removed LNI 10.1.2.3@tcp [ 1290.664048] LNet: Removed LNI 192.168.201.25@tcp [ 1291.191456] Key type .llcrypt unregistered [ 1291.192824] Key type ._llcrypt unregistered [ 1295.422171] Key type ._llcrypt registered [ 1295.424176] Key type .llcrypt registered [ 1295.549728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.565989] alg: No test for adler32 (adler32-zlib) [ 1296.447030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.502148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.524996] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1297.018358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1297.038940] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1297.643404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.25@tcp1 --nid 192.168.201.25@tcp2 [ 1300.417126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp1 192.168.201.25@tcp2 [ 1300.429949] LNetError: 76308:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.25@tcp1 to 192.168.201.25@tcp2 - Route table may be misconfigured [ 1300.444729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp2 192.168.201.25@tcp1 [ 1304.637591] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:35:25 (1789367725) [ 1304.731914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1304.747730] LNet: 76878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.753393] LNetError: 76878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.763457] LNet: Removed LNI 192.168.201.25@tcp1 [ 1305.830714] LNet: Removed LNI 192.168.201.25@tcp2 [ 1305.863892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.920098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.944608] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1306.408094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.423602] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1307.113673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1307.150837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.25@tcp [ 1307.185370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp1 192.168.201.25@tcp1 [ 1307.229568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.25@tcp -d 192.168.201.25@tcp -e network_timeout [ 1307.266720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -e network_timeout [ 1307.304584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -e network_timeout [ 1307.347284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -e network_timeout [ 1307.390558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1307.422099] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1307.438504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1307.532199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1307.639831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1307.936085] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1312.706595] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:35:33 (1789367733) [ 1312.915208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1312.936428] LNet: 77743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.941762] LNetError: 77743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.950952] LNetError: 77743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1314.015355] LNet: Removed LNI 192.168.201.25@tcp [ 1314.020493] LNetError: 77743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.036754] LNet: Removed LNI 192.168.201.25@tcp1 [ 1314.085859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.163654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1314.411417] LNet: 77780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.661530] Key type .llcrypt unregistered [ 1314.666532] Key type ._llcrypt unregistered [ 1323.107598] Key type ._llcrypt registered [ 1323.110608] Key type .llcrypt registered [ 1324.043355] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:35:44 (1789367744) [ 1324.563682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.581612] alg: No test for adler32 (adler32-zlib) [ 1325.461374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.568762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1328.154351] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:35:48 (1789367748) [ 1328.276836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.331391] LNet: 78872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.371379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.448431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.484735] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1328.983780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.999672] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1329.707924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1329.738453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp1 [ 1329.766720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp1 [ 1334.359724] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:35:55 (1789367755) [ 1335.756808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg125-server [ 1337.133376] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:35:57 (1789367757) [ 1338.907833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg125-server [ 1340.395418] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:36:01 (1789367761) [ 1342.028784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg125-server [ 1343.549975] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:36:04 (1789367764) [ 1344.778599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg125-server [ 1345.875120] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:36:06 (1789367766) [ 1347.025516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg125-server [ 1348.324256] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:36:09 (1789367769) [ 1349.598456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg125-server [ 1350.942239] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:36:11 (1789367771) [ 1352.261957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg125-server [ 1353.575833] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:36:14 (1789367774) [ 1354.817693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg125-server [ 1355.975276] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:36:16 (1789367776) [ 1357.115909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg125-server [ 1358.346572] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:36:19 (1789367779) [ 1358.398938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.415976] LNet: 82742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.421179] LNetError: 82742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.426949] LNet: Removed LNI 192.168.201.25@tcp [ 1359.463790] LNet: Removed LNI 192.168.201.25@tcp1 [ 1359.507444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.575193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1359.605874] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1360.369955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1360.460268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1360.593289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1360.609416] LNet: 82897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.612922] LNetError: 82897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.617196] LNetError: 82897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1360.624730] LNet: Removed LNI 192.168.201.25@tcp [ 1360.641905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.696661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1360.715891] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1361.339296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1361.456119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.537152] LNet: Removed LNI 192.168.201.25@tcp [ 1362.566360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.619940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.649723] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1363.233363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1363.317798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1363.457776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.476332] LNet: 83190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.479893] LNet: 83190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1363.486166] LNetError: 83190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.491842] LNetError: 83190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1363.504672] LNet: Removed LNI 192.168.201.25@tcp [ 1363.529658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.568579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1363.589962] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1364.233370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1364.371684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.417927] LNet: Removed LNI 192.168.201.25@tcp [ 1365.434222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.482382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.506346] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1366.057218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1366.123967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1366.262934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.282388] LNetError: 83482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.286401] LNetError: 83482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1366.299608] LNet: Removed LNI 192.168.201.25@tcp [ 1366.310112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.342480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1366.364396] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1366.896917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1367.053077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.102467] LNet: Removed LNI 192.168.201.25@tcp [ 1368.127417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.200894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.227552] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1368.725158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1368.782760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1368.903542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.915729] LNet: 83774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.919715] LNet: 83774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1369.959224] LNet: Removed LNI 192.168.201.25@tcp [ 1369.977805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.025760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1370.040857] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1370.578440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1370.725681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.748566] LNetError: 83920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.758608] LNetError: 83920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1370.777550] LNet: Removed LNI 192.168.201.25@tcp [ 1370.793175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.841545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1370.859969] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1371.340670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1371.399602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1371.568554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.648843] LNet: Removed LNI 192.168.201.25@tcp [ 1372.672869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.739849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1372.758460] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1373.238813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1373.392893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.437743] LNet: Removed LNI 192.168.201.25@tcp [ 1374.456765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.508861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.530719] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1374.934679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1374.986058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1375.125800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1376.165630] LNet: Removed LNI 192.168.201.25@tcp [ 1376.185512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.228141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1376.269157] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1376.789618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1376.934560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1376.943897] LNet: 84505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.948311] LNet: 84505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1378.024649] LNet: Removed LNI 192.168.201.25@tcp [ 1378.044731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.096650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.122583] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1378.615429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1378.663731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.687656] LNet: Removed LNI 192.168.201.25@tcp [ 1378.699744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.732749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.752535] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1379.224946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1379.259686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.275242] LNetError: 84774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.282336] LNetError: 84774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1379.298891] LNet: Removed LNI 192.168.201.25@tcp [ 1379.327549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.366203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1379.418518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.451252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.495499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1379.567484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.592731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.633492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1379.691768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.723340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.777817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1379.797867] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1380.337597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.360680] LNet: Removed LNI 192.168.201.25@tcp [ 1380.374370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.419452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1385.191372] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:36:46 (1789367806) [ 1385.300467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.345399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.389614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1385.414130] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1386.247918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.271117] LNet: Removed LNI 192.168.201.25@tcp231 [ 1386.296933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.362651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1386.393507] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1386.399494] LNet: 85671:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1386.650819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1386.675911] LNet: Removed LNI 192.168.201.25@tcp231 [ 1386.688251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1386.712476] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1387.508578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.537333] LNet: Removed LNI 192.168.201.25@tcp231 [ 1387.556875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.613084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1387.629708] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/0] [ 1387.633731] LNet: 85823:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1387.942305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1387.970538] LNet: Removed LNI 192.168.201.25@tcp231 [ 1387.991335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1388.013290] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/0] [ 1388.720778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.737349] LNet: Removed LNI 192.168.201.25@tcp231 [ 1388.756191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.806513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1388.825611] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/60] [ 1388.830315] LNet: 85974:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1389.144394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1389.193822] LNet: Removed LNI 192.168.201.25@tcp231 [ 1389.220825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1389.253871] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/60] [ 1389.967737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.992739] LNet: Removed LNI 192.168.201.25@tcp231 [ 1390.009554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.075636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1390.101056] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1390.410244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1390.448470] LNet: Removed LNI 192.168.201.25@tcp231 [ 1390.465682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1390.479547] LNet: Added LNI 192.168.201.25@tcp231 [8/256/0/180] [ 1395.254709] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:36:56 (1789367816) [ 1395.333234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.349304] LNet: 86829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.359219] LNet: 86829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1395.369218] LNetError: 86829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.378238] LNetError: 86829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1395.395182] LNet: Removed LNI 192.168.201.25@tcp231 [ 1395.425104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.485477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1395.501089] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 1395.954447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1395.983613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1396.007970] LNet: Removed LNI 192.168.201.25@tcp232 [ 1396.020596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1396.039446] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 1396.570194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1396.598641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1396.627102] LNet: Removed LNI 192.168.201.25@tcp232 [ 1396.643599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1396.664556] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 1397.107336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1397.133535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1397.160521] LNet: Removed LNI 192.168.201.25@tcp232 [ 1397.169611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1397.184724] LNet: Added LNI 192.168.201.25@tcp232 [8/256/0/180] [ 1397.605656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1397.631830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1397.651734] LNet: Removed LNI 192.168.201.25@tcp232 [ 1397.664290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1401.662132] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:37:02 (1789367822) [ 1402.374762] Key type .llcrypt unregistered [ 1402.377171] Key type ._llcrypt unregistered [ 1406.159985] Key type ._llcrypt registered [ 1406.167092] Key type .llcrypt registered [ 1406.350420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.369566] alg: No test for adler32 (adler32-zlib) [ 1407.213220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1407.233428] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1411.561505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1415.874218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.905425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1415.923948] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1416.816474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_interrupt -r 2 [ 1416.844484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 2 [ 1416.866316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_interrupt -r 2 [ 1416.884344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 2 [ 1416.911498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_interrupt -r 2 [ 1416.933269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_interrupt -r 2 [ 1416.963382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_interrupt -r 2 [ 1416.998814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_interrupt -r 2 [ 1417.032624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1417.055635] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1417.061102] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1417.214729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1417.248077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1417.270767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1417.291903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1417.306959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1417.319679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1417.342934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1417.373338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1417.398262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1417.415747] LNet: 88622:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1417.431308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1417.459549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1417.482775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1417.507601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1417.529910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1417.563593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1417.939288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_dropped -r 2 [ 1417.980624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 2 [ 1418.010916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_dropped -r 2 [ 1418.039938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 2 [ 1418.064798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_dropped -r 2 [ 1418.092360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_dropped -r 2 [ 1418.117813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_dropped -r 2 [ 1418.145363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_dropped -r 2 [ 1418.169177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1418.184644] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1418.408857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1418.431939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1418.460493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1418.485606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1418.512468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1418.537863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1418.575566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1418.608894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1418.634598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1418.650680] LNet: 88745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1418.664681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1418.699773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1418.730953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1418.761450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1418.788714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1418.811827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1419.162665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_aborted -r 2 [ 1419.182935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 2 [ 1419.204939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_aborted -r 2 [ 1419.236366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 2 [ 1419.258172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_aborted -r 2 [ 1419.277201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_aborted -r 2 [ 1419.299385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_aborted -r 2 [ 1419.320530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_aborted -r 2 [ 1419.341086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1419.357253] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1419.503291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1419.521201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1419.543968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1419.565686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1419.593486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1419.616337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1419.647232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1419.676662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1419.702353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1419.717664] LNet: 88867:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1419.740942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1419.768811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1419.796136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.823538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.841872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.862199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1420.211135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_no_route -r 2 [ 1420.239351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 2 [ 1420.267322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_no_route -r 2 [ 1420.294548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 2 [ 1420.320233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_no_route -r 2 [ 1420.343323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_no_route -r 2 [ 1420.366889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_no_route -r 2 [ 1420.388407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_no_route -r 2 [ 1420.413929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1420.569165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1420.586760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1420.612817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1420.630781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1420.656774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1420.682275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1420.699782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1420.722148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1420.747849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1420.772694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1420.809827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1420.835975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1420.862909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.891875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1420.915993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1421.221572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e local_timeout -r 2 [ 1421.238708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 2 [ 1421.255890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e local_timeout -r 2 [ 1421.273608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 2 [ 1421.289732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e local_timeout -r 2 [ 1421.311743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e local_timeout -r 2 [ 1421.336565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e local_timeout -r 2 [ 1421.357373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e local_timeout -r 2 [ 1421.381822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1421.397343] LNet: There was an unexpected network error while writing to 192.168.201.125: rc = -22 [ 1421.405438] LNet: Skipped 1 previous similar message [ 1421.536422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1421.553763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1421.578322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1421.598550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1421.615123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1421.641222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1421.661566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1421.688479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1421.709977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1421.720759] LNet: 89111:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1421.725178] LNet: 89111:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1421.734674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1421.752137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1421.775469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.798059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1421.818158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.846751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1422.167493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp -m GET -e remote_dropped -r 2 [ 1422.198886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 2 [ 1422.224656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp -m GET -e remote_dropped -r 2 [ 1422.241753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 2 [ 1422.265914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp -m GET -e remote_dropped -r 2 [ 1422.288742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.125@tcp1 -m GET -e remote_dropped -r 2 [ 1422.313598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp -m GET -e remote_dropped -r 2 [ 1422.354127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.25@tcp1 -d 192.168.201.25@tcp1 -m GET -e remote_dropped -r 2 [ 1422.394550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.25@tcp 192.168.201.125@tcp [ 1422.411407] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1422.581530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1422.601952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1422.624394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1422.654685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1422.679488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1422.704652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1422.730646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1422.759731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1422.781939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1422.803693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1422.831095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1422.864589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1422.891724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1422.913486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1422.940142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1426.536527] LNet: 89307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.540759] LNetError: 89307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.545758] LNet: Removed LNI 192.168.201.25@tcp [ 1426.551925] LNet: Removed LNI 192.168.201.25@tcp1 [ 1426.809779] Key type .llcrypt unregistered [ 1426.812764] Key type ._llcrypt unregistered [ 1437.115110] Key type ._llcrypt registered [ 1437.116040] Key type .llcrypt registered [ 1438.063970] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:37:38 (1789367858) [ 1438.337195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1438.346566] alg: No test for adler32 (adler32-zlib) [ 1439.176959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.218337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.234858] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1439.763605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1439.779107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.25@tcp [ 1439.798175] LNetError: 90282:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1439.861695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.25@tcp [ 1439.929712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1444.310506] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:37:45 (1789367865) [ 1444.480725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.497155] LNet: 90872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.501343] LNetError: 90872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.517969] LNet: Removed LNI 192.168.201.25@tcp [ 1444.547946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.575832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.595217] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1445.027476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.040348] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1445.758560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.25@tcp [ 1445.769770] LNetError: 91138:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.25@tcp to recovery [ 1446.629076] LNet: 1 local NIs in recovery (showing 1): 192.168.201.25@tcp [ 1446.630937] LNet: There was an unexpected network error while writing to 192.168.201.25: rc = -22 [ 1446.999470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.25@tcp [ 1453.846750] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:37:54 (1789367874) [ 1455.171890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg125-server [ 1456.417275] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:37:57 (1789367877) [ 1457.697146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg125-server [ 1458.923345] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:37:59 (1789367879) [ 1460.222891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg125-server [ 1461.336890] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:38:02 (1789367882) [ 1461.443645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.464947] LNet: 92765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.471815] LNetError: 92765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.481267] LNet: Removed LNI 192.168.201.25@tcp [ 1461.496266] LNet: Removed LNI 192.168.201.25@tcp1 [ 1461.510422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.543519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1461.559234] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1462.043512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1462.083199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1462.117240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1462.207834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1462.235313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1462.279273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1466.260768] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:38:07 (1789367887) [ 1466.339582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.365913] LNet: 93493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.372714] LNetError: 93493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.377810] LNetError: 93493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.394963] LNet: Removed LNI 192.168.201.25@tcp [ 1466.410501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.450402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.466221] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1470.549280] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:38:11 (1789367891) [ 1470.636825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.656539] LNet: 94198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.664791] LNetError: 94198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.678196] LNet: Removed LNI 192.168.201.25@tcp [ 1470.688048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.742396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.757183] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1475.001331] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:38:15 (1789367895) [ 1476.027740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1477.365874] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:38:18 (1789367898) [ 1477.781801] LNet: 95234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.787362] LNetError: 95234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.796859] LNet: Removed LNI 192.168.201.25@tcp [ 1478.268543] Key type .llcrypt unregistered [ 1478.270723] Key type ._llcrypt unregistered [ 1482.091195] Key type ._llcrypt registered [ 1482.092742] Key type .llcrypt registered [ 1482.232202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1482.241581] alg: No test for adler32 (adler32-zlib) [ 1483.118238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1483.144528] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1488.010348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1490.773510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.125@tcp [ 1490.958859] LNet: 95647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.964525] LNetError: 95647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.977602] LNet: Removed LNI 192.168.201.25@tcp [ 1491.170529] Key type .llcrypt unregistered [ 1491.171716] Key type ._llcrypt unregistered [ 1497.416147] Key type ._llcrypt registered [ 1497.418110] Key type .llcrypt registered [ 1498.126902] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1498.944754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1499.904910] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:38:41 (1789367921) [ 1500.738971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1501.538377] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:38:42 (1789367922) [ 1502.625583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg125-server [ 1503.578524] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:38:44 (1789367924) [ 1504.545472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg125-server [ 1505.420383] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:38:46 (1789367926) [ 1506.179867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.192618] alg: No test for adler32 (adler32-zlib) [ 1507.099254] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1507.219093] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1508.815210] Key type lgssc registered [ 1509.455534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1520.072046] Key type lgssc unregistered [ 1520.224457] LNet: 98814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.227589] LNetError: 98814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.239350] LNet: Removed LNI 192.168.201.25@tcp [ 1520.584790] Key type .llcrypt unregistered [ 1520.586280] Key type ._llcrypt unregistered [ 1527.996458] Key type ._llcrypt registered [ 1527.998158] Key type .llcrypt registered [ 1528.862605] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:39:09 (1789367949) [ 1529.050759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1529.058387] alg: No test for adler32 (adler32-zlib) [ 1529.935736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.976978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1530.009768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1530.049244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1530.082873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1534.250389] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:39:15 (1789367955) [ 1534.295626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.312234] LNet: 100217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.331949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.384286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.411070] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1535.060266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.071495] LNet: 100368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.074951] LNetError: 100368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.086663] LNet: Removed LNI 192.168.201.25@tcp [ 1535.103402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.140500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1535.166620] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1535.441254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.467763] LNet: Removed LNI 192.168.201.25@tcp [ 1535.484566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.515987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1535.539703] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1539.319840] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:39:20 (1789367960) [ 1539.404704] LNet: 100980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.408602] LNet: 100980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1539.413770] LNetError: 100980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.417847] LNetError: 100980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1539.432055] LNet: Removed LNI 192.168.201.25@tcp [ 1539.929438] Key type .llcrypt unregistered [ 1539.932534] Key type ._llcrypt unregistered [ 1543.083137] Key type ._llcrypt registered [ 1543.085828] Key type .llcrypt registered [ 1543.225144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1543.235326] alg: No test for adler32 (adler32-zlib) [ 1544.076860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.120249] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1544.122329] LNet: *** cfs_fail_loc=e003, val=100*** [ 1544.553210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1544.568995] Ping reply received with mapping: [ 1544.571092] found metadata reply NI[192.168.201.25@tcp] [ 1544.573569] 19 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.575534] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.577671] 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.653862] LNet: 101380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.660452] LNetError: 101380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.674589] LNet: Removed LNI 192.168.201.25@tcp [ 1545.043597] Key type .llcrypt unregistered [ 1545.046542] Key type ._llcrypt unregistered [ 1550.114469] Key type ._llcrypt registered [ 1550.116391] Key type .llcrypt registered [ 1550.758502] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:39:31 (1789367971) [ 1551.062385] Key type .llcrypt unregistered [ 1551.065437] Key type ._llcrypt unregistered [ 1553.769614] Key type ._llcrypt registered [ 1553.772777] Key type .llcrypt registered [ 1553.907404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1553.914537] alg: No test for adler32 (adler32-zlib) [ 1554.833236] LNetError: 102416:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1555.064391] Key type .llcrypt unregistered [ 1555.066607] Key type ._llcrypt unregistered [ 1557.864689] Key type ._llcrypt registered [ 1557.866784] Key type .llcrypt registered [ 1558.536683] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:39:39 (1789367979) [ 1558.844266] Key type .llcrypt unregistered [ 1558.845460] Key type ._llcrypt unregistered [ 1562.395251] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1562.397922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1564.753689] Key type ._llcrypt registered [ 1564.755714] Key type .llcrypt registered [ 1564.890631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.905664] alg: No test for adler32 (adler32-zlib) [ 1565.795372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.834308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1572.441199] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:39:53 (1789367993) [ 1572.515899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.536801] LNet: 104184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.545384] LNetError: 104184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1572.557278] LNet: Removed LNI 10.1.2.3@tcp [ 1572.587292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.089044] Key type .llcrypt unregistered [ 1573.091555] Key type ._llcrypt unregistered [ 1576.624893] Key type ._llcrypt registered [ 1576.626956] Key type .llcrypt registered [ 1576.743606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1576.752659] alg: No test for adler32 (adler32-zlib) [ 1577.648433] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1577.717117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1577.848838] LNet: 104474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.852385] LNetError: 104474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.865439] LNet: Removed LNI 192.168.201.25@tcp [ 1578.263490] Key type .llcrypt unregistered [ 1578.264915] Key type ._llcrypt unregistered [ 1584.754492] Key type ._llcrypt registered [ 1584.756541] Key type .llcrypt registered [ 1585.631244] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:40:06 (1789368006) [ 1586.200423] Key type .llcrypt unregistered [ 1586.203899] Key type ._llcrypt unregistered [ 1589.629552] Key type ._llcrypt registered [ 1589.630830] Key type .llcrypt registered [ 1589.706645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.714058] alg: No test for adler32 (adler32-zlib) [ 1593.846293] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:40:15 (1789368015) [ 1593.921545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.944427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.000737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.021673] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1594.447561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.469764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1594.489713] LNet: Removed LNI 192.168.201.25@tcp [ 1594.607715] LNet: 106211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.825360] Key type .llcrypt unregistered [ 1594.827072] Key type ._llcrypt unregistered [ 1600.935229] Key type ._llcrypt registered [ 1600.936908] Key type .llcrypt registered [ 1601.671886] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:40:22 (1789368022) [ 1601.888721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.898115] alg: No test for adler32 (adler32-zlib) [ 1602.767591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.808957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1602.831574] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1603.313995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1603.347672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.25@tcp [ 1603.364545] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1603.367587] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.370331] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 0 [ 1603.375522] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1603.378404] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1603.381773] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.384650] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 0 [ 1603.387630] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1603.389881] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1603.392764] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.395071] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 1653 [ 1603.397493] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1603.399540] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1603.402071] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.404388] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 1653 [ 1603.408087] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1607.549598] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:40:28 (1789368028) [ 1607.889088] LNet: 107747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.893222] LNetError: 107747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.905145] LNet: Removed LNI 192.168.201.25@tcp [ 1608.399522] Key type .llcrypt unregistered [ 1608.401410] Key type ._llcrypt unregistered [ 1612.616908] Key type ._llcrypt registered [ 1612.618975] Key type .llcrypt registered [ 1612.761105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1612.776895] alg: No test for adler32 (adler32-zlib) [ 1613.635372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1613.653240] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1618.756204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1624.694437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.717249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.732777] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1625.411695] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1625.521590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1625.528898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1625.537656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1627.765465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1627.784909] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1634.579928] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:40:55 (1789368055) [ 1634.644316] LNet: 109245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.651601] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.657275] LNet: Removed LNI 192.168.201.25@tcp [ 1635.679239] LNet: Removed LNI 192.168.201.25@tcp1 [ 1635.683162] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1635.692086] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1635.703894] LNet: Removed LNI 10.1.2.3@tcp99 [ 1636.112608] Key type .llcrypt unregistered [ 1636.114446] Key type ._llcrypt unregistered [ 1640.616213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1640.621746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1643.053287] Key type ._llcrypt registered [ 1643.055213] Key type .llcrypt registered [ 1643.192885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1643.203976] alg: No test for adler32 (adler32-zlib) [ 1644.086344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.120456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.137078] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1644.684838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1644.702308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1646.378495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1646.403023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1646.428797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1646.446344] LNetError: 109992:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1646.475474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1646.509447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1646.807835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1646.854651] LNet: 110012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.858775] LNetError: 110012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1646.862366] LNet: Removed LNI 10.1.2.3@tcp [ 1647.913869] LNet: Removed LNI 192.168.201.25@tcp [ 1648.329427] Key type .llcrypt unregistered [ 1648.331778] Key type ._llcrypt unregistered [ 1655.367820] Key type ._llcrypt registered [ 1655.370374] Key type .llcrypt registered [ 1656.433860] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:41:17 (1789368077) [ 1656.745401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1656.758935] alg: No test for adler32 (adler32-zlib) [ 1657.640537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.680901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.697144] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1658.234446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1658.289054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg125-client@tcp [ 1662.519478] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:41:23 (1789368083) [ 1662.610102] LNet: 111535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.616156] LNetError: 111535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.627731] LNet: Removed LNI 192.168.201.25@tcp [ 1663.250954] Key type .llcrypt unregistered [ 1663.253695] Key type ._llcrypt unregistered [ 1667.851659] Key type ._llcrypt registered [ 1667.853587] Key type .llcrypt registered [ 1667.978937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.996521] alg: No test for adler32 (adler32-zlib) [ 1668.884837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1668.903559] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1668.924708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1673.598806] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:41:34 (1789368094) [ 1674.777438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1676.007561] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:41:36 (1789368096) [ 1676.958333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1678.025521] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:41:39 (1789368099) [ 1678.192669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.204352] LNet: 112962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.208973] LNetError: 112962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.217637] LNet: Removed LNI 192.168.201.25@tcp [ 1678.232475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.263996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.277498] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1678.660503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1678.689798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1683.211654] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:41:44 (1789368104) [ 1683.435460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.460783] LNet: 113671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.475546] LNetError: 113671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.498678] LNet: Removed LNI 192.168.201.25@tcp [ 1683.519715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.560144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1683.598815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1684.067665] Key type .llcrypt unregistered [ 1684.069768] Key type ._llcrypt unregistered [ 1692.080416] Key type ._llcrypt registered [ 1692.082552] Key type .llcrypt registered [ 1692.940662] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:41:53 (1789368113) [ 1693.333762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.347679] alg: No test for adler32 (adler32-zlib) [ 1694.198712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.232553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.258127] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1694.792229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1694.831647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1699.362269] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:42:00 (1789368120) [ 1699.537300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.551659] LNet: 115225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.556304] LNetError: 115225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.567791] LNet: Removed LNI 192.168.201.25@tcp [ 1699.583469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.632334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1699.666430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1703.944553] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:42:04 (1789368124) [ 1704.107865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.122577] LNet: 115805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.147571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.191451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1704.477535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1704.496869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1704.533836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1704.803334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1704.829215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1704.885916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.912542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.971384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1705.184889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1705.204870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1705.235203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1705.440959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1705.459412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1709.168523] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:42:10 (1789368130) [ 1709.326431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.344365] LNet: 116438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.349155] LNet: 116438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1709.365823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.413607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.428460] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1709.872752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1709.887392] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1710.402835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1710.439918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1710.607825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.621391] LNetError: 116718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.626899] LNet: Removed LNI 192.168.201.25@tcp [ 1710.633936] LNet: Removed LNI 192.168.201.25@tcp2 [ 1710.645796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.690746] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1710.693887] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1710.698884] LNet: 116724:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1710.956489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1711.064519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.095163] LNet: Removed LNI 192.168.201.25@tcp [ 1712.098297] LNetError: 116761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.106528] LNetError: 116761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1712.123767] LNet: Removed LNI 192.168.201.25@tcp2 [ 1712.141994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.180300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.197166] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1712.629686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1712.641717] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1713.186910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1713.213692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1713.261792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1713.437475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.453091] LNet: 117045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.458992] LNet: 117045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1714.527368] LNet: Removed LNI 192.168.201.25@tcp [ 1714.529471] LNetError: 117045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.536181] LNetError: 117045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1714.553191] LNet: Removed LNI 192.168.201.25@tcp2 [ 1714.578838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.619350] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1714.624310] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1714.627370] LNet: 117051:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.929605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1715.076838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.104874] LNet: Removed LNI 192.168.201.25@tcp [ 1716.135648] LNet: Removed LNI 192.168.201.25@tcp2 [ 1716.164041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.223277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.255511] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1716.826484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1716.836896] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1717.407271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1717.438273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1717.582880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.602848] LNet: 117368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.607456] LNet: 117368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1718.623178] LNet: Removed LNI 192.168.201.25@tcp [ 1718.625704] LNetError: 117368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.630848] LNetError: 117368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1718.650923] LNet: Removed LNI 192.168.201.25@tcp2 [ 1718.690449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.747161] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1718.754176] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1718.756943] LNet: 117374:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.172601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1719.400691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.479301] LNet: Removed LNI 192.168.201.25@tcp [ 1720.488923] LNet: Removed LNI 192.168.201.25@tcp2 [ 1720.516563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.561078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.580484] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1721.103742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1721.123197] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1721.602873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1721.635522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1721.674124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1721.791711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.847782] LNet: Removed LNI 192.168.201.25@tcp [ 1723.884163] LNet: Removed LNI 192.168.201.25@tcp2 [ 1723.898717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.947295] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1723.952300] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1723.956395] LNet: 117702:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1724.231625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1724.378302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.394731] LNet: Removed LNI 192.168.201.25@tcp [ 1725.414889] LNet: Removed LNI 192.168.201.25@tcp2 [ 1725.432836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.484218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.501721] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1725.933592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1725.949670] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1726.492169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1726.515159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1726.553967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1726.808538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.832209] LNet: 118035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.836705] LNet: 118035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1726.841854] LNetError: 118035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.847643] LNetError: 118035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1726.853680] LNet: Removed LNI 192.168.201.25@tcp [ 1726.862557] LNet: Removed LNI 192.168.201.25@tcp2 [ 1726.880921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.919916] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1726.924639] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1727.186986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1727.434350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.479129] LNet: Removed LNI 192.168.201.25@tcp [ 1728.491613] LNet: Removed LNI 192.168.201.25@tcp2 [ 1728.526478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.572430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.587514] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1729.126050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1729.140664] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1729.762808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1729.796803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1729.824387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1729.868503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1730.088953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.167418] LNet: Removed LNI 192.168.201.25@tcp [ 1732.199790] LNet: Removed LNI 192.168.201.25@tcp2 [ 1732.239531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.290712] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1732.297385] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1732.304695] LNet: 118401:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1732.309427] LNet: 118401:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1732.624528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1736.781272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1737.766768] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:42:38 (1789368158) [ 1737.822541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.838372] LNet: Removed LNI 192.168.201.25@tcp [ 1738.855728] LNet: Removed LNI 192.168.201.25@tcp2 [ 1738.876383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.055401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.077056] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1739.536796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1739.557023] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1740.123743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.138224] LNet: Removed LNI 10.1.2.3@tcp [ 1740.152563] LNet: Removed LNI 192.168.201.25@tcp [ 1740.164922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.209720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1740.226868] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1740.230844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1740.529604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1740.546178] LNet: Removed LNI 192.168.201.25@tcp [ 1740.560407] LNet: Removed LNI 10.1.2.3@tcp [ 1744.592507] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:42:45 (1789368165) [ 1744.680449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.694738] LNet: 120029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.697977] LNet: 120029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1744.723235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.770811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.787320] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1745.230562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1745.249731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1745.276921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.306884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.325798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.347805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.369833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.397245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.415266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.440122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.457809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.477328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1745.567690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1749.394315] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 02:42:50 (1789368170) [ 1749.472781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.506973] LNetError: 120766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.512889] LNetError: 120766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1749.527023] LNet: Removed LNI 192.168.201.25@tcp [ 1749.543373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.585447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.599904] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1750.014479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 1750.033492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1750.054394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.079825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.101613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.134178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.161337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.182302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.209027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.237820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.258899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.277426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.354735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1750.396166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.425610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.445375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.467349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.487326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.507655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.527938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.545586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.566765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1750.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.25@tcp [ 1754.321773] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:42:55 (1789368175) [ 1754.373028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.433645] LNet: Removed LNI 192.168.201.25@tcp [ 1755.455764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.565271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1755.634977] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1755.641814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1755.649065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1757.868477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.897325] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1758.301507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1758.321635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1759.565658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1759.595268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1759.630129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.25@tcp -m PUT -e local_timeout -r 1 [ 1759.676254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1759.714854] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1759.718276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1759.889323] LNetError: 121596:0:(lib-move.c:4794:lnet_parse()) 192.168.201.25@tcp, src 192.168.201.25@tcp: Dropping PUT (error -108 looking up sender) [ 1760.937758] LNet: Removed LNI 10.1.2.3@tcp [ 1761.957824] LNet: Removed LNI 192.168.201.25@tcp [ 1762.173662] Key type .llcrypt unregistered [ 1762.174986] Key type ._llcrypt unregistered [ 1769.124497] Key type ._llcrypt registered [ 1769.126445] Key type .llcrypt registered [ 1770.100119] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:43:10 (1789368190) [ 1770.374624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.383227] alg: No test for adler32 (adler32-zlib) [ 1771.214126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.433750] LNet: 122915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.829569] Key type .llcrypt unregistered [ 1771.831310] Key type ._llcrypt unregistered [ 1775.360694] Key type ._llcrypt registered [ 1775.364027] Key type .llcrypt registered [ 1775.485210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1775.491370] alg: No test for adler32 (adler32-zlib) [ 1776.346050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1776.363645] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1781.015377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1781.113543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1781.421764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.125@tcp [ 1781.471088] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1781.551463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1781.571900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1781.597071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1781.615923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1781.637568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1781.655293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1781.667777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1781.684571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1781.705929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1781.715983] LNet: 123363:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.725612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1781.742818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1781.764688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1781.791833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1781.818241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1781.841513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1782.201967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1782.326889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1782.346788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1782.365920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1782.382642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1782.398745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1782.417639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1782.442603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1782.463339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.487359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1782.508264] LNet: 123482:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.523438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1782.544377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1782.558159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1782.577865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.598608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.623647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1783.044054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.125@tcp [ 1783.220301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1783.246282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1783.271506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1783.296397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1783.319662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1783.340250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1783.357552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1783.375485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1783.402746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1783.425200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1783.446755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1783.464300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1783.486885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1783.505835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1783.522208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1783.877769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1783.998237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1784.012260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1784.029374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1784.046590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1784.061725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1784.083885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1784.104653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1784.124176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1784.142212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1784.151657] LNet: 123720:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.154794] LNet: 123720:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1784.168184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1784.193799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1784.221225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1784.249322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1784.272146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1784.298896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1784.457862] LNet: 123766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.461649] LNetError: 123766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.474779] LNet: Removed LNI 192.168.201.25@tcp [ 1784.714892] Key type .llcrypt unregistered [ 1784.718072] Key type ._llcrypt unregistered [ 1793.402581] Key type ._llcrypt registered [ 1793.407125] Key type .llcrypt registered [ 1794.265164] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:43:35 (1789368215) [ 1794.533577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.552453] alg: No test for adler32 (adler32-zlib) [ 1795.412509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.687357] LNet: 124649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.109641] Key type .llcrypt unregistered [ 1796.111647] Key type ._llcrypt unregistered [ 1799.860740] Key type ._llcrypt registered [ 1799.862602] Key type .llcrypt registered [ 1799.963946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1799.978238] alg: No test for adler32 (adler32-zlib) [ 1800.861378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1800.877614] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1805.745236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.125@tcp [ 1810.989837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.020739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1811.032205] LNet: Added LNI 192.168.201.25@tcp1 [8/256/0/180] [ 1811.544845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1811.934963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.125@tcp [ 1812.116890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.127262] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.125@tcp [ 1812.149704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1812.181228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.206299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1812.230816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1812.248339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1812.269036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.287796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.308431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.326727] LNet: 125298:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.343876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.364288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.384844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.413540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.439996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.462976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.800172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1812.921967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.938954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1812.963409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.982833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.012748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1813.035218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.061811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.094309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.113957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.125243] LNet: 125417:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.136063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.165663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.187335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.209420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.232667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.252681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.617326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.125@tcp [ 1813.760512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.796498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1813.817108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.838516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.865823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1813.891531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.919906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.944480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.965679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.985781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.011237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.045761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.066561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1814.087471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1814.117260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.418720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1814.539511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1814.558131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1814.578668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1814.597262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1814.613427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1814.634653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1814.662453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1814.682274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1814.707209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1814.716655] LNet: 125656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.720915] LNet: 125656:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1814.734611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.751650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.773596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.793128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1814.816149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1814.839905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.024721] LNet: 125736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.032206] LNetError: 125736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.038577] LNet: Removed LNI 192.168.201.25@tcp [ 1818.048247] LNet: Removed LNI 192.168.201.25@tcp1 [ 1818.351793] Key type .llcrypt unregistered [ 1818.355543] Key type ._llcrypt unregistered [ 1829.370985] Key type ._llcrypt registered [ 1829.372256] Key type .llcrypt registered [ 1830.209130] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:44:11 (1789368251) [ 1830.417261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.426125] alg: No test for adler32 (adler32-zlib) [ 1831.268722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.299571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.313570] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1831.762654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1831.778251] LNet: Added LNI 192.168.201.25@tcp2 [8/256/0/180] [ 1832.255267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.25@tcp [ 1832.304149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1833.375149] LNet: Removed LNI 192.168.201.25@tcp [ 1836.973798] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:44:18 (1789368258) [ 1838.031538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1839.047921] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:44:20 (1789368260) [ 1843.142417] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:44:24 (1789368264) [ 1847.100644] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:44:28 (1789368268) [ 1851.329213] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:44:32 (1789368272) [ 1855.200486] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:44:36 (1789368276) [ 1858.974992] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:44:40 (1789368280) [ 1862.945346] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:44:44 (1789368284) [ 1866.834257] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:44:47 (1789368287) [ 1871.250719] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:44:52 (1789368292) [ 1875.481472] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:44:56 (1789368296) [ 1879.403640] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:45:00 (1789368300) [ 1883.039705] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:45:04 (1789368304) [ 1887.273428] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:45:08 (1789368308) [ 1891.436587] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:45:12 (1789368312) [ 1895.681995] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:45:16 (1789368316) [ 1899.490541] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:45:20 (1789368320) [ 1904.082761] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:45:24 (1789368324) [ 1908.806040] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:45:29 (1789368329) [ 1908.917167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.934054] LNet: 137416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.941227] LNetError: 137416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.950084] LNet: Removed LNI 192.168.201.25@tcp2 [ 1908.964603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.033704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.046713] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1909.701443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.714664] LNet: 137558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.718381] LNetError: 137558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.727574] LNet: Removed LNI 192.168.201.25@tcp [ 1909.740124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.770742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1909.785176] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1913.594857] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:45:34 (1789368334) [ 1917.314931] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:45:38 (1789368338) [ 1921.218930] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:45:42 (1789368342) [ 1924.523526] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:45:45 (1789368345) [ 1927.658993] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:45:48 (1789368348) [ 1931.002563] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:45:52 (1789368352) [ 1934.514963] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:45:55 (1789368355) [ 1938.529921] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:45:59 (1789368359) [ 1943.005098] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:46:04 (1789368364) [ 1947.358357] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:46:08 (1789368368) [ 1951.667258] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:46:12 (1789368372) [ 1956.135086] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:46:16 (1789368376) [ 1960.340570] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1771 sec ========== 02:46:21 (1789368381) [ 1960.545282] LNet: 145016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.548674] LNetError: 145016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.558231] LNet: Removed LNI 192.168.201.25@tcp [ 1961.001881] Key type .llcrypt unregistered [ 1961.002961] Key type ._llcrypt unregistered [ 1968.985356] Key type ._llcrypt registered [ 1968.986703] Key type .llcrypt registered [ 1969.298197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1969.309250] alg: No test for adler32 (adler32-zlib) [ 1970.266915] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1970.464019] LNet: Added LNI 192.168.201.25@tcp [8/256/0/180] [ 1972.095527] Key type lgssc registered [ 1972.717739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.116075] Lustre: Mounted lustre-client - version 2.17.58_6_ge9bcf18 [ 2013.488979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2017.586981] Lustre: Unmounted lustre-client [ 2028.959851] Key type lgssc unregistered [ 2029.111571] LNet: 147957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.114675] LNetError: 147957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.124904] LNet: Removed LNI 192.168.201.25@tcp [ 2029.511916] Key type .llcrypt unregistered [ 2029.515744] Key type ._llcrypt unregistered