[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 447046105 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003298] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.023409] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028316] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033480] *** VALIDATE proc *** [ 0.034253] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041014] Speculative Store Bypass: Vulnerable [ 0.044133] debug: unmapping init [mem 0xffffffffa8259000-0xffffffffa8260fff] [ 0.047193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050017] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052015] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056316] rcu: Hierarchical SRCU implementation. [ 0.058466] smp: Bringing up secondary CPUs ... [ 0.059668] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063540] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143020] node 0 deferred pages initialised in 76ms [ 0.146141] devtmpfs: initialized [ 0.147255] x86/mm: Memory block size: 128MB [ 0.149177] gcov: version magic: 0x41383552 [ 0.152273] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158281] pinctrl core: initialized pinctrl subsystem [ 0.160183] [ 0.160969] ************************************************************* [ 0.163016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.168012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170017] ** ** [ 0.173016] ** This means that this kernel is built to expose internal ** [ 0.175013] ** IOMMU data structures, which may compromise security on ** [ 0.178030] ** your system. ** [ 0.180012] ** ** [ 0.182031] ** If you see this message and you are not debugging the ** [ 0.185016] ** kernel, report this immediately to your vendor! ** [ 0.187015] ** ** [ 0.191016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193017] ************************************************************* [ 0.195790] NET: Registered protocol family 16 [ 0.198561] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207209] cpuidle: using governor menu [ 0.208768] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211746] PCI: Using configuration type 1 for base access [ 0.214143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228109] cryptd: max_cpu_qlen set to 1000 [ 0.231285] ACPI: Added _OSI(Module Device) [ 0.233024] ACPI: Added _OSI(Processor Device) [ 0.235020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241048] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247481] ACPI: Interpreter enabled [ 0.249020] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251022] ACPI: Using IOAPIC for interrupt routing [ 0.253137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257707] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268794] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284000] acpiphp: Slot [2] registered [ 0.285088] acpiphp: Slot [5] registered [ 0.287234] acpiphp: Slot [6] registered [ 0.288192] acpiphp: Slot [3] registered [ 0.290136] acpiphp: Slot [4] registered [ 0.292206] acpiphp: Slot [7] registered [ 0.293157] acpiphp: Slot [8] registered [ 0.295158] acpiphp: Slot [9] registered [ 0.297156] acpiphp: Slot [10] registered [ 0.299163] acpiphp: Slot [11] registered [ 0.300218] acpiphp: Slot [12] registered [ 0.302141] acpiphp: Slot [13] registered [ 0.304172] acpiphp: Slot [14] registered [ 0.305167] acpiphp: Slot [15] registered [ 0.307160] acpiphp: Slot [16] registered [ 0.309151] acpiphp: Slot [17] registered [ 0.310151] acpiphp: Slot [18] registered [ 0.312140] acpiphp: Slot [19] registered [ 0.313202] acpiphp: Slot [20] registered [ 0.315210] acpiphp: Slot [21] registered [ 0.317148] acpiphp: Slot [22] registered [ 0.319118] acpiphp: Slot [23] registered [ 0.320129] acpiphp: Slot [24] registered [ 0.322134] acpiphp: Slot [25] registered [ 0.324174] acpiphp: Slot [26] registered [ 0.326165] acpiphp: Slot [27] registered [ 0.328175] acpiphp: Slot [28] registered [ 0.329205] acpiphp: Slot [29] registered [ 0.331157] acpiphp: Slot [30] registered [ 0.333144] acpiphp: Slot [31] registered [ 0.334078] PCI host bridge to bus 0000:00 [ 0.336033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355672] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.359414] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.371053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381927] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390796] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395027] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.405685] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416672] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421967] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446989] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495085] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.498445] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.500376] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.503450] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.506296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.510234] iommu: Default domain type: Passthrough [ 0.512471] SCSI subsystem initialized [ 0.514131] ACPI: bus type USB registered [ 0.515130] usbcore: registered new interface driver usbfs [ 0.518106] usbcore: registered new interface driver hub [ 0.520103] usbcore: registered new device driver usb [ 0.522190] pps_core: LinuxPPS API ver. 1 registered [ 0.523009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.526065] PTP clock support registered [ 0.528108] EDAC MC: Ver: 3.0.0 [ 0.529394] PCI: Using ACPI for IRQ routing [ 0.530658] NetLabel: Initializing [ 0.532017] NetLabel: domain hash size = 128 [ 0.533010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.535109] NetLabel: unlabeled traffic allowed by default [ 0.537187] vgaarb: loaded [ 0.539316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.541017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.547000] clocksource: Switched to clocksource kvm-clock [ 0.655637] VFS: Disk quotas dquot_6.6.0 [ 0.658151] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.660873] *** VALIDATE ramfs *** [ 0.662273] *** VALIDATE hugetlbfs *** [ 0.663819] pnp: PnP ACPI init [ 0.666224] pnp: PnP ACPI: found 6 devices [ 0.682095] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.685767] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.688270] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.690168] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.692373] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.694948] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.698081] NET: Registered protocol family 2 [ 0.700565] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.706518] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.710521] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.715317] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.719080] TCP: Hash tables configured (established 65536 bind 65536) [ 0.721708] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.724397] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.726664] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.730101] NET: Registered protocol family 1 [ 0.733605] RPC: Registered named UNIX socket transport module. [ 0.735901] RPC: Registered udp transport module. [ 0.737855] RPC: Registered tcp transport module. [ 0.739592] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.742297] NET: Registered protocol family 44 [ 0.744241] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.746382] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.748817] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.751049] PCI: CLS 0 bytes, default 64 [ 0.752733] Unpacking initramfs... [ 2.211330] debug: unmapping init [mem 0xffff8919fcc64000-0xffff8919fffcffff] [ 2.215429] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.217927] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.220834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.741934] Initialise system trusted keyrings [ 2.743816] Key type blacklist registered [ 2.745876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.757719] zbud: loaded [ 2.760979] *** VALIDATE nfs *** [ 2.762480] *** VALIDATE nfs4 *** [ 2.764527] pstore: using deflate compression [ 2.769455] Platform Keyring initialized [ 2.881775] NET: Registered protocol family 38 [ 2.883592] Key type asymmetric registered [ 2.885289] Asymmetric key parser 'x509' registered [ 2.887233] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.890455] io scheduler mq-deadline registered [ 2.892243] io scheduler kyber registered [ 2.894129] io scheduler bfq registered [ 2.896108] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.899674] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.902569] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.905493] ACPI: Power Button [PWRF] [ 2.910913] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.917646] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.940521] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.968465] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.996259] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.002480] Non-volatile memory driver v1.3 [ 3.003958] Linux agpgart interface v0.103 [ 3.040211] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.043567] vda: detected capacity change from 0 to 74702848 [ 3.062280] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.065038] vdb: detected capacity change from 0 to 1073741824 [ 3.072550] libphy: Fixed MDIO Bus: probed [ 3.079362] usbcore: registered new interface driver usbserial_generic [ 3.081360] usbserial: USB Serial support registered for generic [ 3.083406] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.087945] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.090100] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.092748] mousedev: PS/2 mouse device common for all mice [ 3.095739] rtc_cmos 00:05: RTC can wake from S4 [ 3.100953] rtc_cmos 00:05: registered as rtc0 [ 3.102659] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.105482] intel_pstate: CPU model not supported [ 3.108896] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.110569] hid: raw HID events driver (C) Jiri Kosina [ 3.117826] usbcore: registered new interface driver usbhid [ 3.118115] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.119992] usbhid: USB HID core driver [ 3.120183] drop_monitor: Initializing network drop monitor service [ 3.127625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.129243] Initializing XFRM netlink socket [ 3.134761] NET: Registered protocol family 10 [ 3.137497] Segment Routing with IPv6 [ 3.139203] NET: Registered protocol family 17 [ 3.141352] mpls_gso: MPLS GSO support [ 3.146798] RAS: Correctable Errors collector initialized. [ 3.149707] AVX version of gcm_enc/dec engaged. [ 3.151461] AES CTR mode by8 optimization enabled [ 3.230165] sched_clock: Marking stable (3230039820, 0)->(4158865980, -928826160) [ 3.234450] registered taskstats version 1 [ 3.236560] Loading compiled-in X.509 certificates [ 3.238797] zswap: loaded using pool lzo/zbud [ 3.268769] Key type big_key registered [ 3.283353] Key type encrypted registered [ 3.285381] ima: No TPM chip found, activating TPM-bypass! [ 3.288165] ima: Allocated hash algorithm: sha1 [ 3.289671] ima: No architecture policies found [ 3.291585] evm: Initialising EVM extended attributes: [ 3.293308] evm: security.selinux [ 3.294367] evm: security.ima [ 3.295329] evm: security.capability [ 3.296492] evm: HMAC attrs: 0x1 [ 3.298694] rtc_cmos 00:05: setting system clock to 2026-09-05 00:58:13 UTC (1788569893) [ 3.305045] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 3.308367] debug: unmapping init [mem 0xffffffffa7f82000-0xffffffffa8258fff] [ 3.317109] Write protecting the kernel read-only data: 28672k [ 3.319887] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.323315] debug: unmapping init [mem 0xffffffffa6f14000-0xffffffffa6ffffff] [ 3.356271] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.365574] systemd[1]: Detected virtualization kvm. [ 3.367171] systemd[1]: Detected architecture x86-64. [ 3.368679] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.394252] systemd[1]: No hostname configured. [ 3.395983] systemd[1]: Set hostname to . [ 3.398232] random: systemd: uninitialized urandom read (16 bytes read) [ 3.400397] systemd[1]: Initializing machine ID from random generator. [ 3.441300] random: ln: uninitialized urandom read (6 bytes read) [ 3.552593] random: systemd: uninitialized urandom read (16 bytes read) [ 3.556393] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.564371] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.569253] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.219250] device-mapper: uevent: version 1.0.3 [ 4.222995] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.979657] virtio_net virtio0 ens2: renamed from eth0 [ 5.075995] scsi host0: ata_piix [ 5.094971] scsi host1: ata_piix [ 5.096959] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.099533] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.590546] dracut-initqueue[582]: RTNETLINK answers: File exists[ 8.594342] random: fast init done [ 9.803915] random: crng init done [ 9.805512] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.139840] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.231563] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.476749] SELinux: Disabled at runtime. [ 11.536041] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.545815] systemd[1]: Detected virtualization kvm. [ 11.548353] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.983721] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.987457] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.993676] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.998190] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.001802] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.009592] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.027862] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ 12.117798] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.432234] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.809659] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.832459] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.967782] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.977320] EDAC sbridge: Ver: 1.1.2 [ 14.255832] Key type dns_resolver registered [ 14.557493] NFS: Registering the id_resolver key type [ 14.559945] Key type id_resolver registered [ 14.561786] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-client login: [ 37.576221] hrtimer: interrupt took 3524355 ns [ 79.999599] libcfs: loading out-of-tree module taints kernel. [ 80.051637] Key type ._llcrypt registered [ 80.083354] Key type .llcrypt registered [ 80.732131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 80.763027] alg: No test for adler32 (adler32-zlib) [ 82.654995] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 83.592972] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 85.391409] Key type lgssc registered [ 87.841549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 229.435347] Lustre: Mounted lustre-client - version 2.17.57_1_g3e0b683 [ 235.238137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.563375] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing check_logdir /tmp/testlogs/ [ 254.952429] Lustre: lustre-OST0000-osc-ffff891a43fbb000: disconnect after 23s idle [ 256.254898] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing yml_node [ 261.759549] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 264.666204] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 268.370297] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 270.587251] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 4 21:02:38 EDT 2026 [ 289.086929] Lustre: Unmounted lustre-client [ 317.041318] Key type lgssc unregistered [ 317.289647] LNet: 8426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.300188] LNetError: 8426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.316963] LNet: Removed LNI 192.168.204.9@tcp [ 318.106722] Key type .llcrypt unregistered [ 318.110945] Key type ._llcrypt unregistered [ 344.782695] Key type ._llcrypt registered [ 344.785321] Key type .llcrypt registered [ 345.031203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.074564] alg: No test for adler32 (adler32-zlib) [ 345.942859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 346.039508] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 346.073863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 350.747858] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 350.841514] LNet: 9308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.854805] LNetError: 9308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.893237] LNet: Removed LNI 192.168.204.9@tcp [ 351.546349] Key type .llcrypt unregistered [ 351.549260] Key type ._llcrypt unregistered [ 358.958622] Key type ._llcrypt registered [ 358.965345] Key type .llcrypt registered [ 360.288421] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:04:08 (1788570248) [ 360.698511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.713675] alg: No test for adler32 (adler32-zlib) [ 361.680963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.795192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.836679] LNet: 9736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.857884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.951324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 361.993035] LNet: 9743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 369.044245] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:04:17 (1788570257) [ 369.300299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.329051] LNet: 10322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.350620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.436200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 376.830386] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:04:25 (1788570265) [ 377.017712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.095539] LNet: 10904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.153991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.290224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 384.876222] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:04:33 (1788570273) [ 385.037877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.077663] LNet: 11492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.099192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.152529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 392.903562] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:04:41 (1788570281) [ 393.091745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.111501] LNet: 12081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.130389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.211636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 393.700296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 401.339702] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:04:50 (1788570290) [ 401.514742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.563489] LNet: 12670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.623738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.730564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.223971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.929984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.265340] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:04:58 (1788570298) [ 410.338503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.394742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.503156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 418.131528] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:05:06 (1788570306) [ 418.471437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.498745] LNet: 13852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.507894] LNet: 13852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 418.540779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.629513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 418.671920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.187856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 419.213751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.661308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 419.707743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 420.215822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 420.267947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 420.734619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 420.792172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.257852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 421.299677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.756177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 421.811964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.332718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 422.373384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.875185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.915762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 423.337193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 423.374184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.850625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.895548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.363576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.414565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.918912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 424.962875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.560131] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:05:20 (1788570320) [ 432.660195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.712529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.849134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 432.892241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 440.343867] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:05:28 (1788570328) [ 440.497387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.644074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.802958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 440.881037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 448.166455] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:05:36 (1788570336) [ 448.693991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.747964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.863485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 448.896987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 457.714610] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:05:46 (1788570346) [ 458.222175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.269545] LNet: 16295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.282679] LNet: 16295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 458.337989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.462420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 458.534129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 466.370034] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:05:54 (1788570354) [ 466.847337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.885767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.978853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 467.015296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 473.687414] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:06:02 (1788570362) [ 473.934807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.995789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.150733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 474.221848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 480.959649] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:06:09 (1788570369) [ 481.280301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.347587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.434436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.811920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.208981] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:06:17 (1788570377) [ 488.361519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.413278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.621674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.887740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.352611] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:06:24 (1788570384) [ 495.554887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.599993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.728637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.944602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.288453] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:06:31 (1788570391) [ 502.644812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.683982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.123218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.401158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.974569] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:06:39 (1788570399) [ 510.338604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.414224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.505279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.579972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.181212] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:06:45 (1788570405) [ 517.433515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.482427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.594702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.654674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 524.275283] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:06:53 (1788570413) [ 524.370262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.392829] LNet: 21733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.403061] LNet: 21733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 524.435995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.500938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 530.391063] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:06:59 (1788570419) [ 530.539617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.598874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.687664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 538.015092] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:07:06 (1788570426) [ 538.359409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.402778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.482616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 539.195742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 546.755847] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:07:15 (1788570435) [ 546.987274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.030843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.103405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 547.168179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 553.538805] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:07:22 (1788570442) [ 553.808879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.852682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.914605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 553.978544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 561.433861] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:07:30 (1788570450) [ 561.600531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.648348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.787247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 561.850732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 568.975104] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:07:37 (1788570457) [ 569.261968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.333034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.489353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 569.567261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 578.156756] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:07:46 (1788570466) [ 578.341855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.456338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.513996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 578.634436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 578.659931] LNetError: 25854:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 578.713927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 578.754353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 585.326708] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:07:54 (1788570474) [ 585.576989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.633074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.761548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZZaHoYOPS5 [ 585.864811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZZaHoYOPS5 [ 585.941322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZZaHoYOPS5 [ 592.937194] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:08:01 (1788570481) [ 593.359416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.420082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.490461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 593.519828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 593.567539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 600.544378] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:08:09 (1788570489) [ 600.773295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.835874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.938695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.963923] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 602.417024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 603.993569] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 606.018704] Key type lgssc registered [ 607.561288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.373917] LNetError: 27616:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.9@tcp99 has no available nets [ 614.220547] Key type lgssc unregistered [ 614.625442] LNetError: 28725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.636563] LNet: Removed LNI 192.168.204.9@tcp [ 615.212569] Key type .llcrypt unregistered [ 615.215261] Key type ._llcrypt unregistered [ 622.414398] Key type ._llcrypt registered [ 622.416280] Key type .llcrypt registered [ 623.705724] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:08:32 (1788570512) [ 624.276962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.314541] alg: No test for adler32 (adler32-zlib) [ 625.231481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.302621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.480705] LNet: Message forwarding will be enabled in 2s [ 631.583267] LNet: Message forwarding enabled [ 632.033555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 632.060223] LNet: Message forwarding will stop in 2s to 6s [ 636.617745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 637.434759] LNet: Message forwarding disabled [ 639.639928] LNet: Message forwarding will be enabled in 2s [ 643.615738] LNet: Message forwarding enabled [ 647.701988] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:08:56 (1788570536) [ 647.875328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.940758] LNet: 30135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.387485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.532158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 655.883880] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:09:04 (1788570544) [ 656.194488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.213759] LNet: 30717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.255954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.341225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 656.388805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 656.449791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 656.503080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 656.531138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 656.564319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 656.601718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.642772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 656.677765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 656.719895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 656.789541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 656.836202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 656.865768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 656.889728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 656.924736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 656.969670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 657.009556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 657.047526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 657.075948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 657.098829] LNetError: 30760:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 663.070739] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:09:11 (1788570551) [ 663.363390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.385664] LNet: 31332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.407519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.485513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.239302] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:09:18 (1788570558) [ 672.287598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg409-server [ 674.731853] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:09:23 (1788570563) [ 674.939932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.970994] LNet: 32264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.998192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.063379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.086774] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 676.131654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[10-14]@tcp [ 676.180398] LNet: 32397:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 676.238307] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.11@tcp is being used as a gateway but routing feature is not turned on [ 676.259381] LNetError: 32268:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.11@tcp has gone from up to down [ 677.229568] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 677.248994] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 678.251021] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 678.277354] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 680.332074] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.10@tcp is being used as a gateway but routing feature is not turned on [ 680.366448] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 683.702917] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:09:32 (1788570572) [ 684.460992] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.11@tcp is being used as a gateway but routing feature is not turned on [ 684.479368] LNetError: 32268:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 685.798203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg409-server [ 688.394455] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:09:36 (1788570576) [ 688.500777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.553230] LNet: 33328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.571978] LNetError: 33328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.617154] LNet: Removed LNI 192.168.204.9@tcp [ 688.642957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.776036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.807439] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 689.865254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 689.893148] LNet: 33452:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.911493] LNet: 33452:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 689.941348] LNetError: 33337:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.12@tcp has gone from up to down [ 689.955259] LNetError: 33337:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 689.980336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 698.600897] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:09:47 (1788570587) [ 698.782348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.825765] LNet: 34037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.844838] LNetError: 34037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.864506] LNet: Removed LNI 192.168.204.9@tcp [ 698.921793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.064508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 699.146778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.187458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.263748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.333755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.380435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.441914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 699.548592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.590681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.659596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.769482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.808930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.869925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 699.979911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.054363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.129087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.230712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.277070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.335652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 700.433091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.500744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.604676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.716229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.753660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.818518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 700.922563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.990634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.066057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.169789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.218667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.335636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 701.432466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.499917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.609702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.453689] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:09:57 (1788570597) [ 710.483540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg409-server [ 712.227036] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:10:01 (1788570601) [ 714.367757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg409-server [ 716.841964] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:10:05 (1788570605) [ 717.018191] LNet: 35440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.030717] LNet: 35440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 717.906393] Key type .llcrypt unregistered [ 717.912567] Key type ._llcrypt unregistered [ 724.768094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.944671] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.959076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.029651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.808456] Key type ._llcrypt registered [ 727.812450] Key type .llcrypt registered [ 728.075445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.105408] alg: No test for adler32 (adler32-zlib) [ 729.052674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.123067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 729.164932] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 730.043693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.066385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.233564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 732.266108] LNet: Removed LNI 10.1.2.3@tcp [ 732.421943] LNet: 36172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.429772] LNetError: 36172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.477702] LNet: Removed LNI 192.168.204.9@tcp [ 733.053125] Key type .llcrypt unregistered [ 733.056762] Key type ._llcrypt unregistered [ 746.727373] Key type ._llcrypt registered [ 746.732627] Key type .llcrypt registered [ 748.162860] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:10:36 (1788570636) [ 748.850608] Key type .llcrypt unregistered [ 748.852904] Key type ._llcrypt unregistered [ 754.611744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.754215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.768580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.784962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.370521] Key type ._llcrypt registered [ 757.378829] Key type .llcrypt registered [ 757.555112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.581215] alg: No test for adler32 (adler32-zlib) [ 758.514746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.612934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.656309] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 759.546846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.578111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.255106] LNet: 37672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.262274] LNetError: 37672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.273305] LNet: Removed LNI 10.1.2.3@tcp [ 762.281540] LNet: Removed LNI 192.168.204.9@tcp [ 762.999889] Key type .llcrypt unregistered [ 763.001646] Key type ._llcrypt unregistered [ 777.030143] Key type ._llcrypt registered [ 777.036689] Key type .llcrypt registered [ 778.427102] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:11:07 (1788570667) [ 778.974723] Key type .llcrypt unregistered [ 778.976573] Key type ._llcrypt unregistered [ 785.703731] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.949272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.965576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.980576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.533445] Key type ._llcrypt registered [ 788.534803] Key type .llcrypt registered [ 788.788093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.825309] alg: No test for adler32 (adler32-zlib) [ 789.738648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.819615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 789.858722] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 792.002735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.017774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.733759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 793.756068] LNet: Removed LNI 10.1.2.3@tcp [ 793.766905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 793.795898] LNet: Removed LNI 10.1.2.31@tcp [ 793.807634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 793.827774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.667095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 794.698141] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 795.625065] LNet: 39599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.635268] LNetError: 39599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 795.643735] LNet: Removed LNI 10.1.2.31@tcp [ 795.657990] LNet: Removed LNI 10.1.2.3@tcp [ 796.400651] Key type .llcrypt unregistered [ 796.403857] Key type ._llcrypt unregistered [ 810.009568] Key type ._llcrypt registered [ 810.015374] Key type .llcrypt registered [ 811.780729] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:11:39 (1788570699) [ 812.333186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.358572] alg: No test for adler32 (adler32-zlib) [ 813.369353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.467391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.523201] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 814.355614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.394252] LNet: 40556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.402825] LNetError: 40556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.417183] LNet: Removed LNI 192.168.204.9@tcp [ 814.435421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.474970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.9@tcp [ 814.494029] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 815.196483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.9@tcp [ 815.226469] LNet: Removed LNI 192.168.204.9@tcp [ 815.305424] LNet: 40691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.027504] Key type .llcrypt unregistered [ 816.031481] Key type ._llcrypt unregistered [ 828.089984] Key type ._llcrypt registered [ 828.093066] Key type .llcrypt registered [ 829.868235] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:11:57 (1788570717) [ 832.098861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg409-server [ 834.264273] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:12:02 (1788570722) [ 834.954916] Key type .llcrypt unregistered [ 834.960232] Key type ._llcrypt unregistered [ 843.178226] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.474674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.485762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.501154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.109903] Key type ._llcrypt registered [ 846.112304] Key type .llcrypt registered [ 846.369976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.398234] alg: No test for adler32 (adler32-zlib) [ 847.437683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.559637] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 848.432499] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 850.558401] LNet: 42537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.562839] LNetError: 42537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.567331] LNet: Removed LNI 192.168.204.9@tcp1 [ 850.575134] LNet: Removed LNI 10.1.2.3@tcp2 [ 851.072541] Key type .llcrypt unregistered [ 851.076398] Key type ._llcrypt unregistered [ 866.586725] Key type ._llcrypt registered [ 866.593915] Key type .llcrypt registered [ 868.795225] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:12:36 (1788570756) [ 869.620439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.656649] alg: No test for adler32 (adler32-zlib) [ 870.678420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.729618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 870.751555] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 872.027131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.067272] LNet: 43512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.075406] LNetError: 43512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.109144] LNet: Removed LNI 192.168.204.9@tcp [ 872.149829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.348107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 872.416881] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 881.515095] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:12:49 (1788570769) [ 882.772933] LNet: 44104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.793536] LNetError: 44104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.835680] LNet: Removed LNI 192.168.204.9@tcp [ 884.063628] Key type .llcrypt unregistered [ 884.066150] Key type ._llcrypt unregistered [ 892.269723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.462895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.472355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.476556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.385900] Key type ._llcrypt registered [ 895.397105] Key type .llcrypt registered [ 895.975872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.001337] alg: No test for adler32 (adler32-zlib) [ 897.070517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.239528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.316108] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 898.607522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 898.636714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.542759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.559859] LNet: 44851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.566701] LNetError: 44851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.572835] LNet: Removed LNI 10.1.2.3@tcp [ 901.586738] LNet: Removed LNI 192.168.204.9@tcp [ 901.621146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.738198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.781118] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 901.786525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.412200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.448439] LNet: 44879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.454477] LNetError: 44879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.463071] LNetError: 44879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 902.474698] LNet: Removed LNI 10.1.2.3@tcp [ 902.484426] LNet: Removed LNI 192.168.204.9@tcp [ 902.527155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.642347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 902.678894] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 902.687094] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.051533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 903.081442] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 904.808349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.838123] LNet: 45043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.855963] LNetError: 45043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.861838] LNetError: 45043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 904.868424] LNet: Removed LNI 10.1.2.3@tcp [ 904.873230] LNet: Removed LNI 192.168.204.9@tcp [ 904.884433] LNet: Removed LNI 192.168.204.9@tcp2 [ 904.935702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.055937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 905.113910] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 905.121822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 905.133654] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 916.305639] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:13:24 (1788570804) [ 916.481861] LNet: 45633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.505808] LNetError: 45633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 916.522707] LNetError: 45633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 916.537517] LNet: Removed LNI 10.1.2.3@tcp [ 916.553332] LNet: Removed LNI 192.168.204.9@tcp [ 916.592584] LNet: Removed LNI 192.168.204.9@tcp2 [ 918.039163] Key type .llcrypt unregistered [ 918.042149] Key type ._llcrypt unregistered [ 926.293092] Key type ._llcrypt registered [ 926.295289] Key type .llcrypt registered [ 926.576561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.608706] alg: No test for adler32 (adler32-zlib) [ 927.668497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.756532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.810289] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 929.143901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.205352] LNet: 46041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.210579] LNetError: 46041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.230231] LNet: Removed LNI 192.168.204.9@tcp [ 929.284322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.401346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.446473] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 930.211784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.250848] LNet: 46071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.258466] LNetError: 46071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.299269] LNet: Removed LNI 192.168.204.9@tcp [ 930.317706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.430045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.473746] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 931.161153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.221388] LNet: Removed LNI 192.168.204.9@tcp [ 931.244477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.357124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 931.389584] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 931.928767] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.192567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.198444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.211548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.664226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.727945] LNet: 46181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.731675] LNet: 46181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 934.738892] LNetError: 46181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.742677] LNetError: 46181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 934.758981] LNet: Removed LNI 192.168.204.9@tcp [ 934.806093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.977822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.019972] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 944.457161] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:13:53 (1788570833) [ 944.604331] LNet: 46773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.617678] LNetError: 46773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.631729] LNet: Removed LNI 192.168.204.9@tcp [ 945.382792] Key type .llcrypt unregistered [ 945.384505] Key type ._llcrypt unregistered [ 952.851711] Key type ._llcrypt registered [ 952.854438] Key type .llcrypt registered [ 953.198813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.220499] alg: No test for adler32 (adler32-zlib) [ 954.237592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.325393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.374116] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 955.303639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.345560] LNet: 47181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.351681] LNetError: 47181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.367940] LNet: Removed LNI 192.168.204.9@tcp [ 955.389502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.468151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 955.501913] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 962.726977] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:14:11 (1788570851) [ 962.810693] LNet: 47764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.824883] LNetError: 47764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.839697] LNet: Removed LNI 192.168.204.9@tcp [ 963.657793] Key type .llcrypt unregistered [ 963.660402] Key type ._llcrypt unregistered [ 969.480484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 969.516451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 969.519821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 969.523617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.090715] Key type ._llcrypt registered [ 972.093219] Key type .llcrypt registered [ 972.286347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.330500] alg: No test for adler32 (adler32-zlib) [ 973.308967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.391444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.442499] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 974.603472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.646216] LNet: 48229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.652406] LNetError: 48229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.670723] LNet: Removed LNI 192.168.204.9@tcp [ 974.703722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.783421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 974.837880] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 983.717712] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:14:31 (1788570871) [ 983.866662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.900391] LNet: 48820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.917934] LNetError: 48820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.935711] LNet: Removed LNI 192.168.204.9@tcp [ 983.996547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.248791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 992.863602] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:14:41 (1788570881) [ 992.952647] LNet: 49402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.615688] Key type .llcrypt unregistered [ 993.620729] Key type ._llcrypt unregistered [ 1001.375337] Key type ._llcrypt registered [ 1001.381389] Key type .llcrypt registered [ 1001.579067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.598553] alg: No test for adler32 (adler32-zlib) [ 1002.539896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.610436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1002.641613] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1003.483743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.504713] LNet: 49813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.509258] LNetError: 49813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.525260] LNet: Removed LNI 192.168.204.9@tcp [ 1003.541203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.639811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1003.693394] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1010.809059] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:14:59 (1788570899) [ 1010.978785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.025191] LNet: 50402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.035620] LNetError: 50402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.053797] LNet: Removed LNI 192.168.204.9@tcp [ 1011.108429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.247855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1018.055729] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:15:07 (1788570907) [ 1019.451900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1021.538849] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:15:10 (1788570910) [ 1023.608837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1025.605870] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:15:14 (1788570914) [ 1027.289545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1028.974273] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:15:17 (1788570917) [ 1030.330786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1032.548406] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:15:21 (1788570921) [ 1034.194984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1035.879607] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:15:24 (1788570924) [ 1036.323327] LNet: 52461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.290900] Key type .llcrypt unregistered [ 1037.292693] Key type ._llcrypt unregistered [ 1044.888788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1045.247917] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1045.267274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1045.271075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1047.970967] Key type ._llcrypt registered [ 1047.973936] Key type .llcrypt registered [ 1048.221596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.254857] alg: No test for adler32 (adler32-zlib) [ 1049.189531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.297214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1049.322293] LNet: Added LNI 192.168.204.9@tcp [8/512/0/180] [ 1049.335221] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1049.337548] LNet: 52783:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1049.469239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1049.496858] LNet: 52802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.504676] LNetError: 52802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1049.508534] LNet: Removed LNI 10.1.2.3@tcp [ 1049.511689] LNetError: 52802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.530479] LNet: Removed LNI 192.168.204.9@tcp [ 1049.570033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.679156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1049.722611] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1049.819623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1049.863737] LNet: 52826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.878066] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.915754] LNet: Removed LNI 192.168.204.9@tcp [ 1049.983731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.122251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1050.158416] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1050.166953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1050.171368] LNet: 52834:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1050.288546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.317862] LNet: 52852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.329816] LNetError: 52852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.336601] LNet: Removed LNI 10.1.2.3@tcp [ 1050.344671] LNetError: 52852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.375061] LNet: Removed LNI 192.168.204.9@tcp [ 1050.398355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.485761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1050.516863] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1050.523914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1050.530118] LNet: 52860:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1050.717747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.735368] LNet: 52886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.746307] LNetError: 52886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.760502] LNet: Removed LNI 10.1.2.3@tcp [ 1050.766186] LNetError: 52886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.787249] LNet: Removed LNI 192.168.204.9@tcp [ 1050.804749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.924951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1050.950527] LNet: Added LNI 192.168.204.9@tcp [8/256/0/182] [ 1050.963379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1051.073265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.101666] LNet: 52914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.114963] LNetError: 52914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1051.120455] LNet: Removed LNI 10.1.2.3@tcp [ 1051.125641] LNetError: 52914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.143831] LNet: Removed LNI 192.168.204.9@tcp [ 1051.173986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.282653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1051.309334] LNet: Added LNI 192.168.204.9@tcp [8/256/0/182] [ 1051.315057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1051.320344] LNet: 52922:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1051.410676] LNet: Removed LNI 10.1.2.3@tcp [ 1051.421269] LNet: Removed LNI 192.168.204.9@tcp [ 1052.200827] Key type .llcrypt unregistered [ 1052.203741] Key type ._llcrypt unregistered [ 1063.819695] Key type ._llcrypt registered [ 1063.824612] Key type .llcrypt registered [ 1065.213472] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:15:53 (1788570953) [ 1066.358599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1068.475873] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:15:56 (1788570956) [ 1068.885454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.908951] alg: No test for adler32 (adler32-zlib) [ 1069.836843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.914743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1069.964640] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1070.530371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.9 [ 1070.545831] LNet: 54210:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1070.552467] LNet: Removed LNI 192.168.204.9@tcp [ 1076.296955] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:16:05 (1788570965) [ 1076.417482] LNet: 54775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.430097] LNetError: 54775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.463200] LNet: Removed LNI 192.168.204.9@tcp [ 1077.099636] Key type .llcrypt unregistered [ 1077.102722] Key type ._llcrypt unregistered [ 1083.765329] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1083.964637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1086.728433] Key type ._llcrypt registered [ 1086.730483] Key type .llcrypt registered [ 1086.899808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.933902] alg: No test for adler32 (adler32-zlib) [ 1088.251328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.804672] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:16:23 (1788570983) [ 1094.948690] LNet: 55689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.958691] LNetError: 55689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1094.984932] LNet: Removed LNI 10.1.2.3@tcp [ 1095.753787] Key type .llcrypt unregistered [ 1095.757166] Key type ._llcrypt unregistered [ 1102.927952] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1103.142709] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1103.146616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1103.160073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1105.843359] Key type ._llcrypt registered [ 1105.849175] Key type .llcrypt registered [ 1106.088231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.107558] alg: No test for adler32 (adler32-zlib) [ 1107.109787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1113.612633] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:16:42 (1788571002) [ 1115.399920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1117.201458] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:16:45 (1788571005) [ 1117.370521] LNet: 56880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.380307] LNetError: 56880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.404368] LNet: Removed LNI 10.1.2.3@tcp [ 1118.266470] Key type .llcrypt unregistered [ 1118.269639] Key type ._llcrypt unregistered [ 1125.007943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1125.216928] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1125.235753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1125.254408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1127.946598] Key type ._llcrypt registered [ 1127.949492] Key type .llcrypt registered [ 1128.181448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.197931] alg: No test for adler32 (adler32-zlib) [ 1129.112743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1129.166918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1138.351839] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:17:06 (1788571026) [ 1138.858515] LNet: 58119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.864664] LNetError: 58119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1138.887402] LNet: Removed LNI 10.1.2.3@tcp [ 1139.641238] Key type .llcrypt unregistered [ 1139.643792] Key type ._llcrypt unregistered [ 1145.372493] Key type ._llcrypt registered [ 1145.375714] Key type .llcrypt registered [ 1145.562283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.579274] alg: No test for adler32 (adler32-zlib) [ 1146.532359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.590909] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1154.396079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1155.173567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1155.220785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1155.272813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1155.288569] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1155.480894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.510656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.534933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.562625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.593566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.638758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.702879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.767719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.792115] LNet: 58560:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.807173] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1155.825698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.886619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.927807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.966860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.006411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.042195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.079654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.466076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1156.519281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1156.574218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1156.598122] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1156.786298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.813543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.843585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.866502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.884947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.905787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.932132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.957529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.971193] LNet: 58669:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.986592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.014288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.051615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.083927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.128822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.172124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.204538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1157.637806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1157.677482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1157.720863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1157.746656] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1157.751991] LNet: Skipped 1 previous similar message [ 1157.970924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.997633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.022577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.071366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.108360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.143478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.204803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.239251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.256782] LNet: 58779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.273916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.299229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.328768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.359681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.396320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.433304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.471820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.956551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1159.002921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1159.044573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1159.392771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.427834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.460984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.496849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.532567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.565848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.620201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.653254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.682347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.709834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.733442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.758745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.785586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.809795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.835646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.272633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1160.313877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1160.346790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1160.362993] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1160.375725] LNet: Skipped 1 previous similar message [ 1160.598799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.641277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.668290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.698683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.723863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.769484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.794240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.814925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.845736] LNet: 58997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.852514] LNet: 58997:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1160.870673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.890502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.925597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.950969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.972982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.994683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.021501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.455560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1161.499035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1161.543962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1161.716440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.742727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.763630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.793816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.828586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.859976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.892755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.916602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.966893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.023679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.074062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.138467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.171954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.205711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.240691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.561596] LNet: 59151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.570896] LNetError: 59151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.626142] LNet: Removed LNI 192.168.204.9@tcp [ 1163.971662] Key type .llcrypt unregistered [ 1163.977480] Key type ._llcrypt unregistered [ 1181.116866] Key type ._llcrypt registered [ 1181.119621] Key type .llcrypt registered [ 1182.565878] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:17:51 (1788571071) [ 1183.383429] Key type .llcrypt unregistered [ 1183.388635] Key type ._llcrypt unregistered [ 1189.781858] Key type ._llcrypt registered [ 1189.784878] Key type .llcrypt registered [ 1189.998552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.022846] alg: No test for adler32 (adler32-zlib) [ 1191.024725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1191.066331] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1199.288178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1211.148850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.194709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1211.221124] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1213.039646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1213.073843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1213.116986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1213.154938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1213.201362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1213.232586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1213.279449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1213.331434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1213.376862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1213.403158] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1213.414370] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1213.671986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.704920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1213.732792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.759841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.784081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.804324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.836821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.861685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.882553] LNet: 60623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.900560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.951816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.019157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.067794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.117590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.170826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.212130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.767872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1214.817832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1214.865640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1214.916690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1214.957593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1214.994842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1215.026781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1215.053307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1215.087564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1215.102563] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1215.108105] LNet: Skipped 2 previous similar messages [ 1215.268392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.293504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.320883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.347617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.381401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.417469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.454374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.482622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.497950] LNet: 60744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.512994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.547535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.577222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.606388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.638315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.677260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.709885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.231544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1216.293970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1216.359833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1216.423454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1216.478225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1216.537000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1216.618436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1216.683173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1216.738962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1216.766170] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1216.772584] LNet: Skipped 2 previous similar messages [ 1217.045545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.091713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.129115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.171420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.218079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.248377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.277992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1217.305545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1217.324476] LNet: 60865:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1217.342157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1217.385231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.435190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.496069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.565576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.637786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.676787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1218.227603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1218.261653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1218.288863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1218.315827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1218.354980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1218.404395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1218.451698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1218.503861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1218.550514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1218.829232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.886335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.921710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.959475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.009824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.069279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.109667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.159170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.210976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.242919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.273835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.303785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.344580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.406424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.454328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.037955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1220.100221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1220.162833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1220.225893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1220.299413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1220.403906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1220.479589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1220.535774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1220.601057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1220.644962] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1220.651994] LNet: Skipped 5 previous similar messages [ 1220.857644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.893975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.940318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.998791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.059712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.111667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.175271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.229649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.246759] LNet: 61108:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.257215] LNet: 61108:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1221.276801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.337157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.384940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.415695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.444265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.475624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.502607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1221.990741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1222.028274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1222.063127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1222.090890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1222.119824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1222.147392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1222.173987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1222.198832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1222.239986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1222.470643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.494409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.520853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.559720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.596502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.646863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.708880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.757409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.787864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.814654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.845605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.872342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.908032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.941121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.987944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.538422] LNet: 61309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.550948] LNetError: 61309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.560947] LNet: Removed LNI 192.168.204.9@tcp [ 1228.571542] LNet: Removed LNI 192.168.204.9@tcp1 [ 1229.022696] Key type .llcrypt unregistered [ 1229.027649] Key type ._llcrypt unregistered [ 1245.423390] Key type ._llcrypt registered [ 1245.425602] Key type .llcrypt registered [ 1246.646909] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:18:55 (1788571135) [ 1247.630311] Key type .llcrypt unregistered [ 1247.636834] Key type ._llcrypt unregistered [ 1254.305232] Key type ._llcrypt registered [ 1254.308318] Key type .llcrypt registered [ 1254.536077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1254.555666] alg: No test for adler32 (adler32-zlib) [ 1255.440895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1255.471544] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1263.908607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1264.764211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1264.823772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1264.887812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1264.927151] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1265.230568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.264654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1265.298198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.334065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1265.377910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.412299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.454586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.476871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.490703] LNet: 62561:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.520571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.562204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.609702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.653232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.710684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.757648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.799105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.315823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1266.358782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1266.390286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1266.408407] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1266.607732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.636765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1266.668463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.712946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.766599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.804451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.842449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.875988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.894356] LNet: 62671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.919649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.962576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.010539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.046733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.088398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.128954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.172716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.614439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1267.648779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1267.699352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1267.729502] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1267.980839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.004420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.027200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.047888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.077991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.104311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.121928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.165860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.191119] LNet: 62780:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.211840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.247616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.278994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.319465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.364371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.401237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.446677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1268.756180] LNet: 62825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.766581] LNetError: 62825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.789082] LNet: Removed LNI 192.168.204.9@tcp [ 1269.199834] Key type .llcrypt unregistered [ 1269.202757] Key type ._llcrypt unregistered [ 1283.836976] Key type ._llcrypt registered [ 1283.839925] Key type .llcrypt registered [ 1285.018312] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:19:33 (1788571173) [ 1285.998478] Key type .llcrypt unregistered [ 1286.001287] Key type ._llcrypt unregistered [ 1291.154097] Key type ._llcrypt registered [ 1291.157946] Key type .llcrypt registered [ 1291.418104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1291.434373] alg: No test for adler32 (adler32-zlib) [ 1292.338493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1292.367977] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1299.455784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1309.324546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1309.377196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1309.414296] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1311.715784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1311.756919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1311.807489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1311.864879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1311.933524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1312.017171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1312.091594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1312.151969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1312.197673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1312.219084] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1312.226985] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1312.501288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1312.534620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1312.570263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1312.596281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.622521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1312.650916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1312.681820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1312.710827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.725836] LNet: 64294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.746117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.805428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.867556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.915739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.986102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.039483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1313.092932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1313.148803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.876690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1313.933977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1313.999797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1314.056657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1314.112867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1314.196387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1314.286352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1314.384777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1314.443651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1314.476235] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1314.480824] LNet: Skipped 2 previous similar messages [ 1314.876611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.960165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1315.008245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.048779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.096083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.147655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.212668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.276931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.319382] LNet: 64418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.372069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.472368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.581047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.672276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.758730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.840486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.951537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.026702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.759401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1316.795935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1316.832959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1316.901695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1316.959883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1317.026270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1317.124875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1317.191603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1317.264938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1317.287623] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1317.295047] LNet: Skipped 1 previous similar message [ 1317.635575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.686135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.730180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.765595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.807350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.852464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.891984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.942902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.964984] LNet: 64541:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.985940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.043758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.110755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.189029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.247404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.310586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.358550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1318.431515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1324.313665] LNet: 64623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.320416] LNetError: 64623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.325866] LNet: Removed LNI 192.168.204.9@tcp [ 1324.341615] LNet: Removed LNI 192.168.204.9@tcp1 [ 1324.737832] Key type .llcrypt unregistered [ 1324.746885] Key type ._llcrypt unregistered [ 1342.442769] Key type ._llcrypt registered [ 1342.444771] Key type .llcrypt registered [ 1343.539627] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:20:32 (1788571232) [ 1345.200863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1347.087624] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:20:36 (1788571236) [ 1348.057841] Key type .llcrypt unregistered [ 1348.062432] Key type ._llcrypt unregistered [ 1355.809483] Key type ._llcrypt registered [ 1355.812322] Key type .llcrypt registered [ 1356.043529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1356.060321] alg: No test for adler32 (adler32-zlib) [ 1356.987388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1357.023337] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1365.154939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1366.125935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1366.154640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1366.213742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1366.238849] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1366.485336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1366.523582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.548687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.572692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.592981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.619768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.643974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.670552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.687605] LNet: 66169:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.708753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.731822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.759102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.785719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.806675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.830596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.061488] LNet: 66213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.066256] LNetError: 66213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.088851] LNet: Removed LNI 192.168.204.9@tcp [ 1367.606609] Key type .llcrypt unregistered [ 1367.610069] Key type ._llcrypt unregistered [ 1389.180305] Key type ._llcrypt registered [ 1389.182435] Key type .llcrypt registered [ 1389.454628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1389.479670] alg: No test for adler32 (adler32-zlib) [ 1390.457448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1390.494035] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1399.086866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1409.750440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.794560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1409.822187] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1411.674637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1411.727395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1411.793827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1411.837841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1411.891808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1411.954323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1412.010200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1412.074597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1412.134901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1412.163537] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1412.534040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1412.567589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1412.608806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.657848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.716571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1412.759660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1412.821628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1412.867931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1412.904572] LNet: 67096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1412.947767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1413.031595] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1413.034179] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1413.036400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1413.042227] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1413.068869] LNet: Skipped 1 previous similar message [ 1413.088180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.123659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1413.170773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1413.216972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1418.475546] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1418.964554] LNet: 67174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.971256] LNetError: 67174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.003273] LNet: Removed LNI 192.168.204.9@tcp [ 1420.005301] LNetError: 67174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.035524] LNet: Removed LNI 192.168.204.9@tcp1 [ 1420.506854] Key type .llcrypt unregistered [ 1420.509493] Key type ._llcrypt unregistered [ 1439.839660] Key type ._llcrypt registered [ 1439.842883] Key type .llcrypt registered [ 1441.111830] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:22:09 (1788571329) [ 1441.486245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1441.506632] alg: No test for adler32 (adler32-zlib) [ 1442.433164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.543035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.574422] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1443.260666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1443.283926] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1444.159577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1444.215722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1444.266457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1444.302908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1444.337942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1444.371147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1444.397064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1444.410748] LNetError: 68275:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1444.575340] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1444.582044] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1446.623244] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1446.768782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1450.720260] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1451.401898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.442581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1451.528979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.571681] LNet: 68382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.585619] LNetError: 68382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.604577] LNet: Removed LNI 192.168.204.9@tcp [ 1451.619425] LNet: Removed LNI 192.168.204.9@tcp1 [ 1451.653700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.747766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.782295] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1452.565452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.579631] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1453.143935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1453.182806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.235084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1453.278396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1453.335498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1453.361909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1453.389074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.417895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1453.439434] LNetError: 68654:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1453.727840] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1456.842457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.871519] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1459.880678] LNet: Skipped 1 previous similar message [ 1464.915580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.955885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1464.980237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1471.801901] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:22:40 (1788571360) [ 1471.994729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.043563] LNet: 69370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.056429] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.069792] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1472.083597] LNet: Removed LNI 192.168.204.9@tcp [ 1472.105657] LNet: Removed LNI 192.168.204.9@tcp1 [ 1472.142301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.225655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.274469] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1473.137729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.151279] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1473.912576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1473.979343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1474.019533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1474.053089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1474.095599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1474.131076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1474.183284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 1474.302981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1475.231385] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.9@tcp [ 1475.232333] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1475.256398] LNet: Skipped 1 previous similar message [ 1489.613920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.766446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1489.801577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.854206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 500 [ 1490.013448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1496.843288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1496.905631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.931371] LNet: 69863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.937601] LNetError: 69863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.945305] LNetError: 69863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1496.950822] LNet: Removed LNI 192.168.204.9@tcp [ 1496.961064] LNet: Removed LNI 192.168.204.9@tcp1 [ 1497.004532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.071543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.094366] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1497.824085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.846487] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1498.641447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1498.668168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1498.716865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1498.743444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1498.774433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1498.801382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1498.828932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1498.853312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 1499.103667] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1499.112105] LNet: Skipped 3 previous similar messages [ 1500.143028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1510.653930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1510.714590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1510.754529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.331647] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:23:27 (1788571407) [ 1519.754990] LNet: 70913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.764905] LNetError: 70913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.770340] LNetError: 70913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.785600] LNet: Removed LNI 192.168.204.9@tcp [ 1519.798340] LNet: Removed LNI 192.168.204.9@tcp1 [ 1520.438197] Key type .llcrypt unregistered [ 1520.440698] Key type ._llcrypt unregistered [ 1528.045837] Key type ._llcrypt registered [ 1528.066232] Key type .llcrypt registered [ 1528.456314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.493260] alg: No test for adler32 (adler32-zlib) [ 1529.531476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1529.579768] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1540.094585] Lustre: DEBUG MARKER: Initial discovery [ 1540.469433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1543.109877] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1543.344779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1543.366771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1543.399817] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1543.430315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1543.490988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -l 3 [ 1543.558881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -e local_error [ 1543.616208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m PUT -l 6 [ 1543.967410] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1545.166421] Lustre: DEBUG MARKER: Force oleg409-client.virtnet to discover 192.168.204.109@tcp (in background) [ 1545.231393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 1546.905808] Lustre: DEBUG MARKER: Force oleg409-server to discover 192.168.204.9@tcp [ 1548.320263] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1549.991279] Lustre: DEBUG MARKER: Wait for 71670 [ 1552.056827] Lustre: DEBUG MARKER: Finished wait on 71670 [ 1552.236566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1552.277957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1552.290060] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1552.597942] LNet: 72132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.609967] LNetError: 72132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.635186] LNet: Removed LNI 192.168.204.9@tcp [ 1553.043559] Key type .llcrypt unregistered [ 1553.050069] Key type ._llcrypt unregistered [ 1572.721774] Key type ._llcrypt registered [ 1572.726587] Key type .llcrypt registered [ 1575.161326] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:24:22 (1788571462) [ 1576.134577] Key type .llcrypt unregistered [ 1576.139287] Key type ._llcrypt unregistered [ 1583.974357] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1584.129709] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1584.132933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1584.135670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1586.857093] Key type ._llcrypt registered [ 1586.867592] Key type .llcrypt registered [ 1587.179513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1587.200883] alg: No test for adler32 (adler32-zlib) [ 1588.083813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.189276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.228234] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1588.898461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1588.914746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1599.485798] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:24:47 (1788571487) [ 1599.739315] LNet: 74210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.749054] LNetError: 74210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1599.759159] LNet: Removed LNI 10.1.2.3@tcp [ 1599.771777] LNet: Removed LNI 192.168.204.9@tcp [ 1600.647139] Key type .llcrypt unregistered [ 1600.656503] Key type ._llcrypt unregistered [ 1607.946418] Key type ._llcrypt registered [ 1607.950641] Key type .llcrypt registered [ 1608.211325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1608.226912] alg: No test for adler32 (adler32-zlib) [ 1609.176944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.222464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.242675] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1609.946875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1609.978486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1611.048276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.9@tcp [ 1611.089400] LNetError: Unexpected error -22 connecting to 192.168.204.9@tcp at host 192.168.204.9:988 [ 1617.294267] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:25:06 (1788571506) [ 1617.505493] LNet: 75345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.512063] LNetError: 75345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1617.520803] LNet: Removed LNI 10.1.2.3@tcp [ 1617.545887] LNet: Removed LNI 192.168.204.9@tcp [ 1618.190473] Key type .llcrypt unregistered [ 1618.193125] Key type ._llcrypt unregistered [ 1624.891902] Key type ._llcrypt registered [ 1624.895295] Key type .llcrypt registered [ 1625.155962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.175324] alg: No test for adler32 (adler32-zlib) [ 1626.173963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.311575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1626.353208] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1627.336643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1627.359194] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1628.303580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.9@tcp1 --nid 192.168.204.9@tcp2 [ 1633.098936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp2 [ 1633.121656] LNetError: 76185:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.9@tcp1 to 192.168.204.9@tcp2 - Route table may be misconfigured [ 1633.153650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp2 192.168.204.9@tcp1 [ 1639.673392] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:25:28 (1788571528) [ 1639.917719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.939871] LNet: 76758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.951470] LNetError: 76758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.957885] LNet: Removed LNI 192.168.204.9@tcp1 [ 1641.000516] LNet: Removed LNI 192.168.204.9@tcp2 [ 1641.065027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.169395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.199863] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1641.908751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.930892] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1642.825542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1642.859768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.9@tcp [ 1642.886125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp1 [ 1642.922931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp -e network_timeout [ 1642.953756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -e network_timeout [ 1642.997967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -e network_timeout [ 1643.040602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -e network_timeout [ 1643.086646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1643.122570] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1643.142866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1643.169771] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1643.275702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1643.408975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1649.871050] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:25:38 (1788571538) [ 1650.164138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.200393] LNet: 77619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.210269] LNetError: 77619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.220687] LNetError: 77619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1650.228497] LNet: Removed LNI 192.168.204.9@tcp [ 1651.304824] LNet: Removed LNI 192.168.204.9@tcp1 [ 1651.366645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.491111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1651.760883] LNet: 77660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.103365] Key type .llcrypt unregistered [ 1652.106212] Key type ._llcrypt unregistered [ 1664.037350] Key type ._llcrypt registered [ 1664.041813] Key type .llcrypt registered [ 1665.677711] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:25:54 (1788571554) [ 1666.159512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.182804] alg: No test for adler32 (adler32-zlib) [ 1667.135892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.651227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1670.420726] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:25:59 (1788571559) [ 1670.638709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.670980] LNet: 78758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.699894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.745761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.764189] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1671.326870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1671.346628] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1671.989644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1672.023912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp1 [ 1672.079943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp1 [ 1679.487736] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:26:07 (1788571567) [ 1681.350301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg409-server [ 1683.462227] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:26:12 (1788571572) [ 1685.920935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg409-server [ 1687.512610] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:26:16 (1788571576) [ 1689.490926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg409-server [ 1691.837862] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:26:20 (1788571580) [ 1694.410341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg409-server [ 1696.982880] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:26:25 (1788571585) [ 1699.026027] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg409-server [ 1701.565693] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:26:30 (1788571590) [ 1704.502660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg409-server [ 1706.918056] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:26:35 (1788571595) [ 1709.197362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg409-server [ 1711.534203] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:26:40 (1788571600) [ 1713.478266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg409-server [ 1715.414409] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:26:44 (1788571604) [ 1718.267137] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg409-server [ 1719.908750] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:26:48 (1788571608) [ 1720.344384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.401708] LNet: 82637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.412163] LNetError: 82637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.422657] LNet: Removed LNI 192.168.204.9@tcp [ 1721.481830] LNet: Removed LNI 192.168.204.9@tcp1 [ 1721.539591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.644216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.672666] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1722.512374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1722.581187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1722.762265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.786330] LNet: 82785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.804088] LNetError: 82785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.811774] LNetError: 82785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1722.825410] LNet: Removed LNI 192.168.204.9@tcp [ 1722.849807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.916983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1722.951432] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1723.796735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1724.024925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.061850] LNetError: 82930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.105601] LNet: Removed LNI 192.168.204.9@tcp [ 1724.143937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.210518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.240916] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1724.951198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1725.033941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1725.266571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.294530] LNet: 83077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.306068] LNet: 83077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1726.376464] LNet: Removed LNI 192.168.204.9@tcp [ 1726.408346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.537787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1726.604897] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1727.617805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1727.863142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.912062] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.919924] LNetError: 83222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1727.943697] LNet: Removed LNI 192.168.204.9@tcp [ 1727.966422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.010670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.036569] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1728.814396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1728.932315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1729.236892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.348192] LNet: Removed LNI 192.168.204.9@tcp [ 1730.394506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.533353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1730.564740] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1731.613842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1731.934527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.983902] LNet: 83515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.998138] LNet: 83515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1732.010037] LNetError: 83515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.016710] LNetError: 83515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1732.032901] LNet: Removed LNI 192.168.204.9@tcp [ 1732.074962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.195060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.230477] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1733.026374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1733.103193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1733.340739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.439145] LNet: Removed LNI 192.168.204.9@tcp [ 1734.495365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.598859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1734.640568] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1735.531931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1737.037628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.097692] LNet: Removed LNI 192.168.204.9@tcp [ 1738.159783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.284853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.314058] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1739.338532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1739.424766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1739.690110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.776734] LNet: Removed LNI 192.168.204.9@tcp [ 1740.829508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.984663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1741.071523] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1742.031233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1742.430522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.476846] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.483747] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1742.502505] LNetError: 84110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.511488] LNetError: 84110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1742.534044] LNet: Removed LNI 192.168.204.9@tcp [ 1742.568499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.690901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.765901] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1743.920815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1744.038726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1744.402599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.511819] LNet: Removed LNI 192.168.204.9@tcp [ 1745.555075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.669618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1745.714971] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1746.741417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 1747.181236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.263746] LNet: Removed LNI 192.168.204.9@tcp [ 1748.354903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.420884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.447328] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1749.457886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1749.536341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.588987] LNet: Removed LNI 192.168.204.9@tcp [ 1749.618246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.717894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.789201] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1750.621701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1750.697217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.730099] LNet: Removed LNI 192.168.204.9@tcp [ 1750.751116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.840574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1750.915940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.955463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.016799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1751.146954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.194443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.245841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1751.326449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.357674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.417532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1751.447555] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1752.261251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.316359] LNet: Removed LNI 192.168.204.9@tcp [ 1752.343958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.429871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1761.731624] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:27:29 (1788571649) [ 1761.909118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.963350] LNet: 85419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.970519] LNet: 85419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1762.010256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.113823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1762.147040] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1763.380887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.407578] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.417662] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1763.441903] LNet: Removed LNI 192.168.204.9@tcp231 [ 1763.473465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.534357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1763.569170] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1763.579364] LNet: 85571:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.205935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1764.269435] LNet: Removed LNI 192.168.204.9@tcp231 [ 1764.289705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1764.328851] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1765.647147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.685093] LNet: Removed LNI 192.168.204.9@tcp231 [ 1765.715233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.808454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1765.842391] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 1765.848399] LNet: 85723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.676598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1766.746832] LNet: Removed LNI 192.168.204.9@tcp231 [ 1766.764261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1766.835854] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 1768.402600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.460809] LNet: Removed LNI 192.168.204.9@tcp231 [ 1768.481890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.559204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1768.579577] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 1768.584167] LNet: 85875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.931559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1768.968284] LNet: Removed LNI 192.168.204.9@tcp231 [ 1768.986459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1769.027753] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 1770.229288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.275060] LNet: Removed LNI 192.168.204.9@tcp231 [ 1770.294529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.425203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1770.457382] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1771.362814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1771.416431] LNet: Removed LNI 192.168.204.9@tcp231 [ 1771.430433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1771.463255] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 1781.542929] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:27:49 (1788571669) [ 1781.844490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.961154] LNet: Removed LNI 192.168.204.9@tcp231 [ 1782.002123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.076564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1782.100720] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1782.865988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1782.907107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1782.942545] LNet: Removed LNI 192.168.204.9@tcp232 [ 1782.974565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1783.019495] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1783.742391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1783.773452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1783.804539] LNet: Removed LNI 192.168.204.9@tcp232 [ 1783.819503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1783.851426] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1784.623751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1784.654443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1784.686458] LNet: Removed LNI 192.168.204.9@tcp232 [ 1784.714854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1784.753200] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 1785.608942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1785.663354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1785.694970] LNet: Removed LNI 192.168.204.9@tcp232 [ 1785.722601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1792.543472] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:28:01 (1788571681) [ 1794.751375] Key type .llcrypt unregistered [ 1794.755128] Key type ._llcrypt unregistered [ 1802.378773] Key type ._llcrypt registered [ 1802.388753] Key type .llcrypt registered [ 1802.766351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1802.791251] alg: No test for adler32 (adler32-zlib) [ 1803.715196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1803.744430] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1812.498325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1822.262690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.303557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1822.347463] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1824.025329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 1824.069870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 1824.138987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 1824.173902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 1824.236279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 1824.291747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 1824.336189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 1824.379413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 1824.444269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1824.497798] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1824.502561] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1824.725888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1824.767373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1824.806821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1824.874252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1824.932300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1824.970964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1825.020784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1825.059552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1825.076912] LNet: 88534:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1825.099336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1825.130561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1825.165528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1825.194427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1825.237067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1825.274081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1825.811859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 1825.856155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 1825.891488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 1825.928952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 1825.976708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 1826.007784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 1826.041873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 1826.077597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 1826.117122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1826.140803] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1826.339592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1826.368252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1826.392816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1826.419144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1826.446941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1826.476425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1826.502856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1826.529975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1826.544855] LNet: 88654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.563740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1826.592868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1826.624636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1826.656635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1826.686909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1826.716036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1827.207732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 1827.245706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 1827.293110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 1827.345192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 1827.382305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 1827.416924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 1827.452698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 1827.483909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 1827.510939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1827.528709] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1827.737135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1827.761644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1827.788663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.819950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.854593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.884367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.904825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.933702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.957151] LNet: 88774:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.986652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1828.017975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1828.060811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1828.119193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1828.165890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1828.226308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.718310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 1828.764421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 1828.807525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 1828.857441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 1828.916319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 1828.963415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 1829.003117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 1829.034669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 1829.068262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1829.369379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1829.400578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1829.427218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1829.451768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1829.493621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.533387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.563211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.594606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.625670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.652092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.682403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.721316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.775967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.840969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1830.486416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 1830.530368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 1830.576566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 1830.613913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 1830.649944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 1830.689890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 1830.733822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 1830.785323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 1830.834588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1830.853927] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1830.860548] LNet: Skipped 1 previous similar message [ 1831.103798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1831.138964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1831.169392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.208237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.244830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.273326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.300931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.332880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.349966] LNet: 89015:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.354924] LNet: 89015:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1831.370147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.398334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.436672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.462859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.497441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.545871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.181712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 1832.231659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 1832.278570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 1832.316785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 1832.364520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 1832.399088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 1832.441467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 1832.487427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 1832.532666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 1832.560096] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1832.745554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1832.774194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1832.800220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1832.825207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1832.851456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1832.876508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1832.908215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1832.941495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1832.989449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.020780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.055538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.112585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.174173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.222198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.800858] LNet: 89213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.806141] LNetError: 89213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.813327] LNet: Removed LNI 192.168.204.9@tcp [ 1837.819660] LNet: Removed LNI 192.168.204.9@tcp1 [ 1838.267701] Key type .llcrypt unregistered [ 1838.272781] Key type ._llcrypt unregistered [ 1855.204133] Key type ._llcrypt registered [ 1855.211258] Key type .llcrypt registered [ 1857.009709] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:29:05 (1788571745) [ 1857.379802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.396230] alg: No test for adler32 (adler32-zlib) [ 1858.324539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.379311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.409649] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1859.406492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1859.460951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1859.478565] LNetError: 90194:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1859.543733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.9@tcp [ 1859.650171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.899971] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:29:14 (1788571754) [ 1866.240979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.279997] LNet: 90784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.289793] LNetError: 90784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.305904] LNet: Removed LNI 192.168.204.9@tcp [ 1866.320164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.361821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1866.383134] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1867.010536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1867.030128] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1867.911978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.9@tcp [ 1867.938156] LNetError: 91050:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1868.383409] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1868.387826] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1869.261205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.9@tcp [ 1877.829683] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:29:26 (1788571766) [ 1879.415113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg409-server [ 1881.603366] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:29:30 (1788571770) [ 1883.351021] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg409-server [ 1885.012975] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:29:33 (1788571773) [ 1887.117177] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg409-server [ 1889.199235] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:29:37 (1788571777) [ 1889.318445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.346135] LNet: 92665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.354885] LNetError: 92665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.363440] LNet: Removed LNI 192.168.204.9@tcp [ 1889.377495] LNet: Removed LNI 192.168.204.9@tcp1 [ 1889.416766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.500669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.527815] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1890.162995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1890.218364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1890.260867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1890.321836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1890.357970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1890.386432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1897.794531] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:29:46 (1788571786) [ 1898.082968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.122460] LNet: 93399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.127781] LNetError: 93399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.136380] LNetError: 93399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1898.163301] LNet: Removed LNI 192.168.204.9@tcp [ 1898.192536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.239735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.260177] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1905.904840] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:29:54 (1788571794) [ 1906.166809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.194062] LNet: 94103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.216908] LNetError: 94103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.243719] LNet: Removed LNI 192.168.204.9@tcp [ 1906.262626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.331722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.375032] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1913.299466] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:30:01 (1788571801) [ 1914.958736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1916.955602] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:30:05 (1788571805) [ 1917.743275] LNet: 95138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.752422] LNetError: 95138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.767135] LNet: Removed LNI 192.168.204.9@tcp [ 1918.522581] Key type .llcrypt unregistered [ 1918.532244] Key type ._llcrypt unregistered [ 1926.067387] Key type ._llcrypt registered [ 1926.073895] Key type .llcrypt registered [ 1926.250293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1926.263171] alg: No test for adler32 (adler32-zlib) [ 1927.184793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1927.215911] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1935.375927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1941.244906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.109@tcp [ 1941.708184] LNet: 95565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.722885] LNetError: 95565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.738154] LNet: Removed LNI 192.168.204.9@tcp [ 1942.280720] Key type .llcrypt unregistered [ 1942.282392] Key type ._llcrypt unregistered [ 1954.950931] Key type ._llcrypt registered [ 1954.953260] Key type .llcrypt registered [ 1956.302150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1958.164591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1960.579962] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:30:48 (1788571848) [ 1962.279607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1963.958767] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:30:52 (1788571852) [ 1965.861562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg409-server [ 1967.635888] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:30:56 (1788571856) [ 1969.812858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg409-server [ 1971.974691] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:31:00 (1788571860) [ 1973.456505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1973.611952] alg: No test for adler32 (adler32-zlib) [ 1974.747355] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1975.121349] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1976.855165] Key type lgssc registered [ 1977.840158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1997.202285] Key type lgssc unregistered [ 1997.469601] LNet: 98744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.486559] LNetError: 98744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.511422] LNet: Removed LNI 192.168.204.9@tcp [ 1998.185848] Key type .llcrypt unregistered [ 1998.189933] Key type ._llcrypt unregistered [ 2014.465149] Key type ._llcrypt registered [ 2014.471725] Key type .llcrypt registered [ 2016.114272] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:31:44 (1788571904) [ 2016.410173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.423205] alg: No test for adler32 (adler32-zlib) [ 2017.393166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.454811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2017.510719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2017.577295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2017.627847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.109668] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:31:53 (1788571913) [ 2024.568227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.587985] LNet: 100161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.607865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.664815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.710469] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2025.580737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.601669] LNet: 100305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.612974] LNetError: 100305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.642934] LNet: Removed LNI 192.168.204.9@tcp [ 2025.684512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.740187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2025.769111] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2026.391968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2026.412618] LNetError: 100333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.427729] LNet: Removed LNI 192.168.204.9@tcp [ 2026.482906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.544174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2026.568724] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2033.316831] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:32:02 (1788571922) [ 2033.452415] LNet: 100920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.463357] LNet: 100920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2033.471883] LNetError: 100920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.514688] LNet: Removed LNI 192.168.204.9@tcp [ 2034.200601] Key type .llcrypt unregistered [ 2034.207456] Key type ._llcrypt unregistered [ 2040.338245] Key type ._llcrypt registered [ 2040.342269] Key type .llcrypt registered [ 2040.610279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.641175] alg: No test for adler32 (adler32-zlib) [ 2041.535719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.598323] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2041.602143] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.282179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2042.307437] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.311782] LNet: Skipped 2 previous similar messages [ 2042.324842] Ping reply received with mapping: [ 2042.328592] found metadata reply NI[192.168.204.9@tcp] [ 2042.335900] 09 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.340951] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.344044] 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.499439] LNet: 101324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.507786] LNetError: 101324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.525561] LNet: Removed LNI 192.168.204.9@tcp [ 2043.185798] Key type .llcrypt unregistered [ 2043.190193] Key type ._llcrypt unregistered [ 2054.323127] Key type ._llcrypt registered [ 2054.327568] Key type .llcrypt registered [ 2055.556759] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:32:24 (1788571944) [ 2056.288821] Key type .llcrypt unregistered [ 2056.290597] Key type ._llcrypt unregistered [ 2063.342193] Key type ._llcrypt registered [ 2063.343844] Key type .llcrypt registered [ 2063.573956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.592352] alg: No test for adler32 (adler32-zlib) [ 2064.870301] LNetError: 102383:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2065.320604] Key type .llcrypt unregistered [ 2065.326820] Key type ._llcrypt unregistered [ 2070.317769] Key type ._llcrypt registered [ 2070.320270] Key type .llcrypt registered [ 2071.832742] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:32:40 (1788571960) [ 2072.531626] Key type .llcrypt unregistered [ 2072.538544] Key type ._llcrypt unregistered [ 2079.908404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2079.921343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2082.494974] Key type ._llcrypt registered [ 2082.500972] Key type .llcrypt registered [ 2082.784197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2082.800255] alg: No test for adler32 (adler32-zlib) [ 2083.745347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.833057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2094.103068] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:33:03 (1788571983) [ 2094.422508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.440482] LNet: 104171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.448679] LNetError: 104171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2094.460074] LNet: Removed LNI 10.1.2.3@tcp [ 2094.477243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.289179] Key type .llcrypt unregistered [ 2095.292809] Key type ._llcrypt unregistered [ 2102.994449] Key type ._llcrypt registered [ 2102.996723] Key type .llcrypt registered [ 2103.201925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2103.237101] alg: No test for adler32 (adler32-zlib) [ 2104.169112] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2104.253581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2104.404736] LNet: 104464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.417805] LNetError: 104464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.439293] LNet: Removed LNI 192.168.204.9@tcp [ 2105.336461] Key type .llcrypt unregistered [ 2105.339833] Key type ._llcrypt unregistered [ 2117.432338] Key type ._llcrypt registered [ 2117.433962] Key type .llcrypt registered [ 2118.856189] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:33:27 (1788572007) [ 2119.922384] Key type .llcrypt unregistered [ 2119.924726] Key type ._llcrypt unregistered [ 2127.811424] Key type ._llcrypt registered [ 2127.814043] Key type .llcrypt registered [ 2128.148806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2128.184399] alg: No test for adler32 (adler32-zlib) [ 2137.337699] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:33:46 (1788572026) [ 2137.599352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.660632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.751552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.785597] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2138.436567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.490886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2138.535376] LNet: Removed LNI 192.168.204.9@tcp [ 2138.885203] LNet: 106222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.383268] Key type .llcrypt unregistered [ 2139.387156] Key type ._llcrypt unregistered [ 2152.693798] Key type ._llcrypt registered [ 2152.696746] Key type .llcrypt registered [ 2154.476962] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:34:02 (1788572042) [ 2154.921611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.947065] alg: No test for adler32 (adler32-zlib) [ 2155.859938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.938895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.980802] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2156.786175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2156.843185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.9@tcp [ 2156.861506] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2156.872791] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.876767] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 0 [ 2156.885205] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2156.889932] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2156.894469] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.900886] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 0 [ 2156.905996] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2156.908760] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2156.912679] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.915416] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 2206 [ 2156.919635] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2156.922262] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2156.932257] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2156.941312] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2156, 0, 2206 [ 2156.946414] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2163.895917] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:34:12 (1788572052) [ 2164.425658] LNet: 107763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.431534] LNetError: 107763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.449229] LNet: Removed LNI 192.168.204.9@tcp [ 2165.096883] Key type .llcrypt unregistered [ 2165.098739] Key type ._llcrypt unregistered [ 2171.346099] Key type ._llcrypt registered [ 2171.352135] Key type .llcrypt registered [ 2171.557456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2171.575970] alg: No test for adler32 (adler32-zlib) [ 2172.512714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2172.548169] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2181.005521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2190.962304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.023922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2191.054197] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2192.262650] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2192.487036] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2192.491976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2192.495614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2194.946671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2194.982124] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2206.252664] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:34:54 (1788572094) [ 2206.455952] LNet: 109270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.470083] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.482567] LNet: Removed LNI 192.168.204.9@tcp [ 2207.525818] LNet: Removed LNI 192.168.204.9@tcp1 [ 2207.535039] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2207.552435] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2207.574486] LNet: Removed LNI 10.1.2.3@tcp99 [ 2208.322719] Key type .llcrypt unregistered [ 2208.327263] Key type ._llcrypt unregistered [ 2215.316539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2215.319819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2217.894893] Key type ._llcrypt registered [ 2217.897977] Key type .llcrypt registered [ 2218.173387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2218.192792] alg: No test for adler32 (adler32-zlib) [ 2219.175805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.241610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.266202] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2219.879215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2219.903854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2222.018706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2222.073645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2222.125571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2222.154831] LNetError: 110020:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2222.187552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2222.243621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2222.790662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2222.876724] LNet: 110041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.888462] LNetError: 110041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2222.894333] LNet: Removed LNI 10.1.2.3@tcp [ 2223.910738] LNet: Removed LNI 192.168.204.9@tcp [ 2224.655574] Key type .llcrypt unregistered [ 2224.657384] Key type ._llcrypt unregistered [ 2238.102300] Key type ._llcrypt registered [ 2238.104144] Key type .llcrypt registered [ 2239.589650] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:35:28 (1788572128) [ 2240.173055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.203965] alg: No test for adler32 (adler32-zlib) [ 2241.152912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.225513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.246707] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2242.037595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2242.134411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg409-client@tcp [ 2249.536920] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:35:38 (1788572138) [ 2249.629272] LNet: 111571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.635281] LNetError: 111571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.648512] LNet: Removed LNI 192.168.204.9@tcp [ 2250.457870] Key type .llcrypt unregistered [ 2250.463703] Key type ._llcrypt unregistered [ 2258.182952] Key type ._llcrypt registered [ 2258.185564] Key type .llcrypt registered [ 2258.494984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.523322] alg: No test for adler32 (adler32-zlib) [ 2259.500968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2259.578674] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2259.646906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2267.506891] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:35:56 (1788572156) [ 2269.129469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2271.321600] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:36:00 (1788572160) [ 2273.104790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2274.886899] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:36:03 (1788572163) [ 2275.131529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2275.153969] LNet: 113005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.159389] LNetError: 113005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.168994] LNet: Removed LNI 192.168.204.9@tcp [ 2275.184489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.256744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.290873] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2276.059706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2276.095699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2282.739620] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:36:11 (1788572171) [ 2283.018388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.043402] LNet: 113714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.054419] LNetError: 113714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.069877] LNet: Removed LNI 192.168.204.9@tcp [ 2284.105396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.195404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2284.250934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2284.645620] LNet: 113754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.068364] Key type .llcrypt unregistered [ 2285.071534] Key type ._llcrypt unregistered [ 2297.786449] Key type ._llcrypt registered [ 2297.788076] Key type .llcrypt registered [ 2299.399682] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:36:27 (1788572187) [ 2299.785054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.800207] alg: No test for adler32 (adler32-zlib) [ 2300.794968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.855319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.895431] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2301.871215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2301.975935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2309.500681] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:36:38 (1788572198) [ 2309.597728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.637496] LNet: 115268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.647462] LNetError: 115268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.674610] LNet: Removed LNI 192.168.204.9@tcp [ 2309.714719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.800080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2309.851718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2316.160900] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:36:45 (1788572205) [ 2316.296240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.334780] LNet: 115849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.359201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.459837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2316.813788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.840875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.896703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2317.308394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2317.356601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2317.436230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.456884] LNet: 115887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.495604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.555959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2318.096341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2318.124361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2318.182624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2318.602642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2318.647949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2325.650160] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:36:54 (1788572214) [ 2325.778788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.826181] LNet: 116482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.876190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.963087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.018245] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2327.009159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2327.032681] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2327.688520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2327.728343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2327.928111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.947585] LNetError: 116766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.954177] LNet: Removed LNI 192.168.204.9@tcp [ 2327.964942] LNet: Removed LNI 192.168.204.9@tcp2 [ 2327.981750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.028370] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2328.034630] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2328.040419] LNet: 116772:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2328.586735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2328.880678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.951192] LNet: Removed LNI 192.168.204.9@tcp [ 2329.955249] LNetError: 116809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.960337] LNetError: 116809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2329.989116] LNet: Removed LNI 192.168.204.9@tcp2 [ 2330.042625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.100718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.123127] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2330.724443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2330.745621] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2331.467202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2331.500446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2331.541718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2331.728762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.748548] LNet: 117094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.754244] LNet: 117094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2332.767198] LNet: Removed LNI 192.168.204.9@tcp [ 2332.770212] LNetError: 117094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.777402] LNetError: 117094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2332.795768] LNet: Removed LNI 192.168.204.9@tcp2 [ 2332.821494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.903476] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2332.915520] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2332.922630] LNet: 117100:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.375537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2333.567375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.582583] LNet: Removed LNI 192.168.204.9@tcp [ 2334.629970] LNet: Removed LNI 192.168.204.9@tcp2 [ 2334.668127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.734694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.766198] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2335.414444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2335.437241] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2336.321486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2336.382957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2336.596524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.632991] LNet: Removed LNI 192.168.204.9@tcp [ 2337.638228] LNetError: 117417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.652727] LNetError: 117417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2337.679682] LNet: Removed LNI 192.168.204.9@tcp2 [ 2337.728936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.826475] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2337.832589] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2337.842864] LNet: 117423:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.252524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2338.471351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.551178] LNet: Removed LNI 192.168.204.9@tcp [ 2339.577237] LNet: Removed LNI 192.168.204.9@tcp2 [ 2339.606263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.695939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2339.740278] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2340.434130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2340.452108] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2341.265729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2341.306701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2341.367121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2341.528510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.545896] LNet: 117745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.552217] LNet: 117745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2342.559192] LNet: Removed LNI 192.168.204.9@tcp [ 2343.599293] LNet: Removed LNI 192.168.204.9@tcp2 [ 2343.669523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.769616] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2343.779279] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2343.791638] LNet: 117751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.350798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2344.516697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.532700] LNet: Removed LNI 192.168.204.9@tcp [ 2345.576946] LNet: Removed LNI 192.168.204.9@tcp2 [ 2345.608176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.682660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.704655] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2346.349688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2346.370422] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2347.431804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2347.467138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2347.547399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2347.938431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.982769] LNetError: 118084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.998398] LNetError: 118084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2348.009353] LNet: Removed LNI 192.168.204.9@tcp [ 2348.029141] LNet: Removed LNI 192.168.204.9@tcp2 [ 2348.061958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.122450] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2348.127459] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2348.131484] LNet: 118090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.746272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2349.135912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.175673] LNet: Removed LNI 192.168.204.9@tcp [ 2350.187972] LNet: Removed LNI 192.168.204.9@tcp2 [ 2350.215105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.282120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.305995] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2351.173632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2351.200248] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2351.941105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2351.984202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2352.008235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2352.054967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2352.304607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.375176] LNet: Removed LNI 192.168.204.9@tcp [ 2354.407961] LNet: Removed LNI 192.168.204.9@tcp2 [ 2354.450544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.519750] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2354.529267] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2355.088408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2361.553359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2363.445589] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:37:32 (1788572252) [ 2363.546489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.610889] LNet: 119204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.623671] LNet: 119204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2363.634199] LNet: Removed LNI 192.168.204.9@tcp [ 2363.661590] LNet: Removed LNI 192.168.204.9@tcp2 [ 2363.721378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.125508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.153687] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2364.958700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2364.980341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2365.768961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.788833] LNetError: 119483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2365.794950] LNetError: 119483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2365.801867] LNet: Removed LNI 10.1.2.3@tcp [ 2365.813982] LNet: Removed LNI 192.168.204.9@tcp [ 2365.843605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.910966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2365.964139] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2365.971205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2366.454395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2366.495984] LNet: Removed LNI 192.168.204.9@tcp [ 2366.512765] LNet: Removed LNI 10.1.2.3@tcp [ 2372.220439] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:37:41 (1788572261) [ 2372.283535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.323854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.386618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2372.416178] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2373.052742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2373.086204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.119639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.147375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.170903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.199364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.226506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.251714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.278495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.313457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.348550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2373.488651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2379.315159] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:37:48 (1788572268) [ 2379.460863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.517530] LNet: Removed LNI 192.168.204.9@tcp [ 2380.555551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.710265] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2380.820143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.828378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2380.841290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2383.331141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.355898] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2383.957253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2383.987977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2387.055483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2387.123324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2387.245838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.9@tcp -m PUT -e local_timeout -r 1 [ 2387.376209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2387.469717] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2387.478483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2387.830071] LNetError: 120892:0:(lib-move.c:4801:lnet_parse()) 192.168.204.9@tcp, src 192.168.204.9@tcp: Dropping PUT (error -108 looking up sender) [ 2388.840890] LNet: Removed LNI 10.1.2.3@tcp [ 2389.863575] LNet: Removed LNI 192.168.204.9@tcp [ 2390.300432] Key type .llcrypt unregistered [ 2390.302546] Key type ._llcrypt unregistered [ 2402.421900] Key type ._llcrypt registered [ 2402.427442] Key type .llcrypt registered [ 2403.736335] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:38:12 (1788572292) [ 2404.065822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.088585] alg: No test for adler32 (adler32-zlib) [ 2404.997380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.375830] LNet: 122207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.963889] Key type .llcrypt unregistered [ 2405.967751] Key type ._llcrypt unregistered [ 2412.612225] Key type ._llcrypt registered [ 2412.618110] Key type .llcrypt registered [ 2412.771593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2412.792992] alg: No test for adler32 (adler32-zlib) [ 2413.755790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2413.795551] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2421.181722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2421.372335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2421.898700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 2421.990413] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 2422.167567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.192282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.214765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.247515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.291720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.336823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.395606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.433842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.465166] LNet: 122660:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.485431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.529159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.553680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.603739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.658384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.721295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.348317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2423.575264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2423.597360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2423.628548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2423.660473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2423.684072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2423.712522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2423.737687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2423.765260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2423.782463] LNet: 122778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.819346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2423.867417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.916636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.955782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.999406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.040563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.704981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 2424.953307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.975172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.002497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.027866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.053599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.096466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.167089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.213388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.245286] LNet: 122895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2425.274384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.318225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.357659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.396975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.431840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.479162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2426.252291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2426.520344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2426.546189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2426.587714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2426.635176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2426.677454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2426.707338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2426.749301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2426.790957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2426.855601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2426.907442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2426.955865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2426.998424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2427.043671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2427.070602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.348446] LNet: 123062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.361624] LNetError: 123062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.382438] LNet: Removed LNI 192.168.204.9@tcp [ 2427.823228] Key type .llcrypt unregistered [ 2427.827502] Key type ._llcrypt unregistered [ 2444.010877] Key type ._llcrypt registered [ 2444.017426] Key type .llcrypt registered [ 2445.393278] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:38:53 (1788572333) [ 2445.856643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2445.869793] alg: No test for adler32 (adler32-zlib) [ 2446.802660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.163464] LNet: 123950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.725390] Key type .llcrypt unregistered [ 2447.727521] Key type ._llcrypt unregistered [ 2453.736390] Key type ._llcrypt registered [ 2453.742244] Key type .llcrypt registered [ 2453.980237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2454.010884] alg: No test for adler32 (adler32-zlib) [ 2455.003547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2455.027322] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2462.724376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2471.468705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.525125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2471.550572] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2472.306774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2472.843387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 2473.037045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2473.063335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2473.086955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2473.115991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2473.155335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2473.190593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2473.235274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2473.265068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2473.283223] LNet: 124606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2473.298874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2473.335539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2473.385438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2473.426225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2473.468625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2473.497926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2474.006537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2474.209595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2474.234872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2474.265705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2474.297946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2474.326361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2474.353767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2474.389085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2474.432650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2474.459216] LNet: 124723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2474.480903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2474.513477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2474.532645] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.109@tcp, 192.168.204.109@tcp1 [ 2474.569455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2474.608360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2474.644101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2474.686035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2475.366432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 2475.628981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2475.678709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2475.717182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2475.755432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2475.795963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2475.823921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2475.855182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2475.905601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2475.939332] LNet: 124840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2475.967437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2476.025641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2476.060242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2476.088886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2476.122552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2476.158970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2476.821978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2477.070345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2477.098859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2477.130128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2477.164816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2477.200524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.243787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.275874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2477.318252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2477.353299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2477.395113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2477.429797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2477.469589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2477.500306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2477.533500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.419713] LNet: 125042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2482.428272] LNetError: 125042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2482.438386] LNet: Removed LNI 192.168.204.9@tcp [ 2482.449086] LNet: Removed LNI 192.168.204.9@tcp1 [ 2482.901520] Key type .llcrypt unregistered [ 2482.906250] Key type ._llcrypt unregistered [ 2499.439810] Key type ._llcrypt registered [ 2499.441966] Key type .llcrypt registered [ 2501.285387] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:39:49 (1788572389) [ 2502.336908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.375036] alg: No test for adler32 (adler32-zlib) [ 2503.401819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.486313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2503.523221] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2504.728076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2504.759808] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2506.360123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2506.489214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2507.551155] LNet: Removed LNI 192.168.204.9@tcp [ 2514.236469] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:40:03 (1788572403) [ 2515.596972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2517.154110] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:40:06 (1788572406) [ 2524.656780] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:40:13 (1788572413) [ 2532.484527] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:40:20 (1788572420) [ 2540.335212] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:40:29 (1788572429) [ 2548.573351] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:40:37 (1788572437) [ 2556.124583] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:40:44 (1788572444) [ 2562.969451] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:40:51 (1788572451) [ 2571.047558] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:40:59 (1788572459) [ 2577.562771] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:41:06 (1788572466) [ 2584.542405] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:41:13 (1788572473) [ 2591.422960] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:41:20 (1788572480) [ 2599.213543] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:41:27 (1788572487) [ 2606.499703] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:41:35 (1788572495) [ 2613.091078] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:41:42 (1788572502) [ 2619.711173] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:41:48 (1788572508) [ 2626.059979] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:41:55 (1788572515) [ 2634.596251] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:42:03 (1788572523) [ 2641.141721] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:42:10 (1788572530) [ 2641.386850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2641.408720] LNet: 136745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2641.413756] LNetError: 136745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.475684] LNet: Removed LNI 192.168.204.9@tcp2 [ 2642.510858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2642.644109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2642.676805] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2643.665990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2643.690480] LNet: 136888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.702447] LNetError: 136888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2643.718401] LNet: Removed LNI 192.168.204.9@tcp [ 2643.735104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2643.809599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2643.841242] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2649.871720] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:42:18 (1788572538) [ 2656.680562] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:42:25 (1788572545) [ 2663.067804] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:42:32 (1788572552) [ 2670.143486] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:42:38 (1788572558) [ 2676.427219] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:42:45 (1788572565) [ 2683.446539] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:42:52 (1788572572) [ 2690.958950] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:42:59 (1788572579) [ 2698.566604] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:43:07 (1788572587) [ 2706.035376] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:43:15 (1788572595) [ 2714.279779] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:43:22 (1788572602) [ 2722.549797] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:43:31 (1788572611) [ 2728.660289] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:43:37 (1788572617) [ 2735.214333] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2463 sec ========== 21:43:44 (1788572624) [ 2735.584846] LNet: 144355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.591515] LNetError: 144355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.614619] LNet: Removed LNI 192.168.204.9@tcp [ 2736.602728] Key type .llcrypt unregistered [ 2736.609782] Key type ._llcrypt unregistered [ 2750.217741] Key type ._llcrypt registered [ 2750.220813] Key type .llcrypt registered [ 2750.659054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2750.713488] alg: No test for adler32 (adler32-zlib) [ 2751.757821] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2751.945756] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2753.616086] Key type lgssc registered [ 2754.685954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2816.164856] Lustre: Mounted lustre-client - version 2.17.57_1_g3e0b683 [ 2821.136989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2830.553871] Lustre: Unmounted lustre-client [ 2854.158672] Key type lgssc unregistered [ 2854.514068] LNet: 147324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.530222] LNetError: 147324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.549920] LNet: Removed LNI 192.168.204.9@tcp [ 2855.230138] Key type .llcrypt unregistered [ 2855.234412] Key type ._llcrypt unregistered