[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454444949 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002010] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.006516] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012065] *** VALIDATE selinux *** [ 0.020259] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024759] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] *** VALIDATE tmpfs *** [ 0.028409] *** VALIDATE proc *** [ 0.030043] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035020] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039283] debug: unmapping init [mem 0xffffffffb1c59000-0xffffffffb1c60fff] [ 0.041142] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042543] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043018] ... version: 2 [ 0.043858] ... bit width: 48 [ 0.044013] ... generic registers: 4 [ 0.044871] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046012] ... fixed-purpose events: 3 [ 0.047008] ... event mask: 000000070000000f [ 0.048300] rcu: Hierarchical SRCU implementation. [ 0.050373] smp: Bringing up secondary CPUs ... [ 0.051581] x86: Booting SMP configuration: [ 0.052021] .... node #0, CPUs: #1 #2 #3 [ 0.055501] smp: Brought up 1 node, 4 CPUs [ 0.057013] smpboot: Max logical packages: 1 [ 0.058019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135924] node 0 deferred pages initialised in 75ms [ 0.139100] devtmpfs: initialized [ 0.140276] x86/mm: Memory block size: 128MB [ 0.144182] gcov: version magic: 0x41383552 [ 0.148322] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155299] pinctrl core: initialized pinctrl subsystem [ 0.157174] [ 0.157831] ************************************************************* [ 0.161024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163020] ** ** [ 0.166014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169012] ** ** [ 0.171014] ** This means that this kernel is built to expose internal ** [ 0.174018] ** IOMMU data structures, which may compromise security on ** [ 0.178014] ** your system. ** [ 0.180013] ** ** [ 0.183011] ** If you see this message and you are not debugging the ** [ 0.185016] ** kernel, report this immediately to your vendor! ** [ 0.188024] ** ** [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193022] ************************************************************* [ 0.197742] NET: Registered protocol family 16 [ 0.199615] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211118] cpuidle: using governor menu [ 0.213000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216532] PCI: Using configuration type 1 for base access [ 0.218223] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233031] cryptd: max_cpu_qlen set to 1000 [ 0.235208] ACPI: Added _OSI(Module Device) [ 0.237014] ACPI: Added _OSI(Processor Device) [ 0.239013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246449] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252320] ACPI: Interpreter enabled [ 0.254057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255021] ACPI: Using IOAPIC for interrupt routing [ 0.257120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273817] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293450] acpiphp: Slot [2] registered [ 0.295107] acpiphp: Slot [5] registered [ 0.297116] acpiphp: Slot [6] registered [ 0.299116] acpiphp: Slot [3] registered [ 0.301122] acpiphp: Slot [4] registered [ 0.303098] acpiphp: Slot [7] registered [ 0.305097] acpiphp: Slot [8] registered [ 0.307158] acpiphp: Slot [9] registered [ 0.309110] acpiphp: Slot [10] registered [ 0.311094] acpiphp: Slot [11] registered [ 0.313081] acpiphp: Slot [12] registered [ 0.314057] acpiphp: Slot [13] registered [ 0.315073] acpiphp: Slot [14] registered [ 0.316071] acpiphp: Slot [15] registered [ 0.317069] acpiphp: Slot [16] registered [ 0.318054] acpiphp: Slot [17] registered [ 0.319082] acpiphp: Slot [18] registered [ 0.320055] acpiphp: Slot [19] registered [ 0.321052] acpiphp: Slot [20] registered [ 0.321997] acpiphp: Slot [21] registered [ 0.322968] acpiphp: Slot [22] registered [ 0.324080] acpiphp: Slot [23] registered [ 0.325105] acpiphp: Slot [24] registered [ 0.326057] acpiphp: Slot [25] registered [ 0.327055] acpiphp: Slot [26] registered [ 0.328064] acpiphp: Slot [27] registered [ 0.328978] acpiphp: Slot [28] registered [ 0.330052] acpiphp: Slot [29] registered [ 0.331056] acpiphp: Slot [30] registered [ 0.332022] acpiphp: Slot [31] registered [ 0.332964] PCI host bridge to bus 0000:00 [ 0.334011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350251] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352983] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356184] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363515] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.365455] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374111] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376664] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380802] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388801] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.394015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.407015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413728] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420596] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465592] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507714] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.510469] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.513367] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.515464] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.518344] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.522676] iommu: Default domain type: Passthrough [ 0.526426] SCSI subsystem initialized [ 0.528125] ACPI: bus type USB registered [ 0.530154] usbcore: registered new interface driver usbfs [ 0.533089] usbcore: registered new interface driver hub [ 0.535080] usbcore: registered new device driver usb [ 0.538176] pps_core: LinuxPPS API ver. 1 registered [ 0.540010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.544065] PTP clock support registered [ 0.547084] EDAC MC: Ver: 3.0.0 [ 0.549181] PCI: Using ACPI for IRQ routing [ 0.550892] NetLabel: Initializing [ 0.553109] NetLabel: domain hash size = 128 [ 0.555010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.557081] NetLabel: unlabeled traffic allowed by default [ 0.559413] vgaarb: loaded [ 0.561394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.563014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.571684] clocksource: Switched to clocksource kvm-clock [ 0.686771] VFS: Disk quotas dquot_6.6.0 [ 0.688332] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.690868] *** VALIDATE ramfs *** [ 0.692117] *** VALIDATE hugetlbfs *** [ 0.693660] pnp: PnP ACPI init [ 0.695942] pnp: PnP ACPI: found 6 devices [ 0.715445] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.717837] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.719514] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.721148] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.722774] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.724626] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.726693] NET: Registered protocol family 2 [ 0.728650] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.732643] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.735274] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.741375] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.745752] TCP: Hash tables configured (established 65536 bind 65536) [ 0.749505] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.753458] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.757097] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760906] NET: Registered protocol family 1 [ 0.763792] RPC: Registered named UNIX socket transport module. [ 0.766728] RPC: Registered udp transport module. [ 0.769196] RPC: Registered tcp transport module. [ 0.771550] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.774712] NET: Registered protocol family 44 [ 0.776482] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778551] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780995] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.784453] PCI: CLS 0 bytes, default 64 [ 0.787687] Unpacking initramfs... [ 2.157600] debug: unmapping init [mem 0xffff95237cc64000-0xffff95237ffcffff] [ 2.161735] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.164171] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.167215] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.643149] Initialise system trusted keyrings [ 2.644996] Key type blacklist registered [ 2.647067] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.656606] zbud: loaded [ 2.659798] *** VALIDATE nfs *** [ 2.661165] *** VALIDATE nfs4 *** [ 2.662939] pstore: using deflate compression [ 2.666620] Platform Keyring initialized [ 2.744748] NET: Registered protocol family 38 [ 2.746430] Key type asymmetric registered [ 2.747635] Asymmetric key parser 'x509' registered [ 2.748942] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.752360] io scheduler mq-deadline registered [ 2.753803] io scheduler kyber registered [ 2.755177] io scheduler bfq registered [ 2.756446] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.759222] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.762115] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.764653] ACPI: Power Button [PWRF] [ 2.769628] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.775723] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.784593] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.810205] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.838979] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.843918] Non-volatile memory driver v1.3 [ 2.845371] Linux agpgart interface v0.103 [ 2.878298] virtio_blk virtio1: [vda] 134784 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.880966] vda: detected capacity change from 0 to 69009408 [ 2.900804] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.904534] vdb: detected capacity change from 0 to 1073741824 [ 2.911833] libphy: Fixed MDIO Bus: probed [ 2.922447] usbcore: registered new interface driver usbserial_generic [ 2.925190] usbserial: USB Serial support registered for generic [ 2.927658] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.931401] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.932759] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.935293] mousedev: PS/2 mouse device common for all mice [ 2.937649] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.939438] rtc_cmos 00:05: RTC can wake from S4 [ 2.944788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.947886] rtc_cmos 00:05: registered as rtc0 [ 2.950050] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.950456] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.956162] intel_pstate: CPU model not supported [ 2.959345] hid: raw HID events driver (C) Jiri Kosina [ 2.961692] usbcore: registered new interface driver usbhid [ 2.963847] usbhid: USB HID core driver [ 2.965423] drop_monitor: Initializing network drop monitor service [ 2.967945] Initializing XFRM netlink socket [ 2.969979] NET: Registered protocol family 10 [ 2.973102] Segment Routing with IPv6 [ 2.974596] NET: Registered protocol family 17 [ 2.976861] mpls_gso: MPLS GSO support [ 2.983117] RAS: Correctable Errors collector initialized. [ 2.984840] AVX version of gcm_enc/dec engaged. [ 2.986213] AES CTR mode by8 optimization enabled [ 3.063393] sched_clock: Marking stable (3063368638, 0)->(3830844660, -767476022) [ 3.067336] registered taskstats version 1 [ 3.069125] Loading compiled-in X.509 certificates [ 3.071023] zswap: loaded using pool lzo/zbud [ 3.097996] Key type big_key registered [ 3.113659] Key type encrypted registered [ 3.115496] ima: No TPM chip found, activating TPM-bypass! [ 3.117623] ima: Allocated hash algorithm: sha1 [ 3.119336] ima: No architecture policies found [ 3.121175] evm: Initialising EVM extended attributes: [ 3.123111] evm: security.selinux [ 3.124369] evm: security.ima [ 3.125481] evm: security.capability [ 3.127088] evm: HMAC attrs: 0x1 [ 3.129546] rtc_cmos 00:05: setting system clock to 2026-05-16 10:09:36 UTC (1778926176) [ 3.136619] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.139905] debug: unmapping init [mem 0xffffffffb1982000-0xffffffffb1c58fff] [ 3.151077] Write protecting the kernel read-only data: 28672k [ 3.154624] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.157836] debug: unmapping init [mem 0xffffffffb0914000-0xffffffffb09fffff] [ 3.190373] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.198500] systemd[1]: Detected virtualization kvm. [ 3.200441] systemd[1]: Detected architecture x86-64. [ 3.202247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.227672] systemd[1]: No hostname configured. [ 3.229125] systemd[1]: Set hostname to . [ 3.230806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.232769] systemd[1]: Initializing machine ID from random generator. [ 3.371576] random: systemd: uninitialized urandom read (16 bytes read) [ 3.374172] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.378040] random: systemd: uninitialized urandom read (16 bytes read) [ 3.380400] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.384146] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.957573] device-mapper: uevent: version 1.0.3 [ 3.959704] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.300668] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.040520] virtio_net virtio0 ens2: renamed from eth0 [ 5.446447] scsi host0: ata_piix [ 5.606529] scsi host1: ata_piix [ 5.612637] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.625595] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.640573] random: crng init done [ 10.646246] random: 7 urandom warning(s) missed due to ratelimiting [ 12.890607] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.410353] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.659909] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.039190] SELinux: Disabled at runtime. [ 19.232795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.263085] systemd[1]: Detected virtualization kvm. [ 19.269196] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.176329] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.180581] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.201897] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.212763] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.232812] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.258634] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.286173] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Apply Kernel Variables. [ 22.119545] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.216630] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.480383] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.783534] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.865653] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.171989] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 31.893413] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 32.795207] NFS: Registering the id_resolver key type [ 32.802338] Key type id_resolver registered [ 32.807140] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-client login: [ 106.829787] libcfs: loading out-of-tree module taints kernel. [ 107.117041] Key type ._llcrypt registered [ 107.121427] Key type .llcrypt registered [ 108.056396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 108.085156] alg: No test for adler32 (adler32-zlib) [ 108.445011] hrtimer: interrupt took 3928762 ns [ 109.972895] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 111.108651] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 112.897371] Key type lgssc registered [ 115.359455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 262.929289] Lustre: Mounted lustre-client [ 268.187154] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.910938] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing check_logdir /tmp/testlogs/ [ 287.818975] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing yml_node [ 288.736422] Lustre: lustre-OST0000-osc-ffff9523c64cf800: disconnect after 24s idle [ 291.757042] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 295.140283] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 298.344178] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 299.902956] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat May 16 06:14:32 EDT 2026 [ 320.040304] LustreError: 7935:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9523c64cf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 320.095818] Lustre: Unmounted lustre-client [ 349.667326] Key type lgssc unregistered [ 349.938736] LNet: 8381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.944851] LNetError: 8381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 349.958302] LNet: Removed LNI 192.168.202.40@tcp [ 350.791201] Key type .llcrypt unregistered [ 350.792909] Key type ._llcrypt unregistered [ 377.228873] Key type ._llcrypt registered [ 377.231167] Key type .llcrypt registered [ 377.402100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 377.434655] alg: No test for adler32 (adler32-zlib) [ 378.307408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 378.367851] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 378.385440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 382.495410] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 382.679546] LNet: 9261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.708154] LNetError: 9261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.727842] LNet: Removed LNI 192.168.202.40@tcp [ 383.297245] Key type .llcrypt unregistered [ 383.299534] Key type ._llcrypt unregistered [ 390.247899] Key type ._llcrypt registered [ 390.256902] Key type .llcrypt registered [ 391.941021] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:16:03 (1778926563) [ 392.372515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 392.397301] alg: No test for adler32 (adler32-zlib) [ 393.347268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.443874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.476457] LNet: 9676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.503210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.575483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 393.599852] LNet: 9683:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 400.913014] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:16:12 (1778926572) [ 401.043561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.070612] LNet: 10256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.106824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.176787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 408.278927] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:16:20 (1778926580) [ 408.386204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.425800] LNet: 10842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.455891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.579638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 415.285440] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:16:27 (1778926587) [ 415.463787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.513925] LNet: 11431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.551795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.640485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 423.292258] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:16:35 (1778926595) [ 423.495250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.543784] LNet: 12015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.566620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.693627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 424.190629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 431.714184] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:16:43 (1778926603) [ 431.828876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.871384] LNet: 12601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.907726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.994801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 432.556875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 433.011545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 439.793223] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:16:51 (1778926611) [ 440.059677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.158819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.273908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 446.491465] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:16:58 (1778926618) [ 446.742615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.796091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.841704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 446.878825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 447.366627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 447.401530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 447.824273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 447.899649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.373130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 448.412555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 448.833796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 448.905863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.396520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 449.451264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 449.914525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 449.947989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 450.459585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 450.492856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 450.923513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 450.964973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 451.526034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 451.601526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 452.278612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 452.318464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 452.948926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 452.980916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 453.477556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 453.530557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 461.220513] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:17:12 (1778926632) [ 461.322192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.344096] LNet: 14465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.346528] LNet: 14465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 461.384158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.556724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 461.578478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 468.508801] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:17:20 (1778926640) [ 468.779969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.830243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.916341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 468.949962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 476.207362] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:17:28 (1778926648) [ 476.567508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.612335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.755725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 476.834615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 485.129546] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:17:36 (1778926656) [ 485.605239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.701620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.818359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 485.873081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 493.650249] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:17:45 (1778926665) [ 493.852688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.895402] LNet: 16816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.906274] LNet: 16816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 493.942895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.020468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 494.057477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 501.160225] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:17:53 (1778926673) [ 501.642969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.676675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.781184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 501.836958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 509.106511] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:18:00 (1778926680) [ 509.245406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.324370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.430566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.114885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.696107] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:18:10 (1778926690) [ 518.804255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.857186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.089996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.252763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.020129] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:18:17 (1778926697) [ 526.113898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.207868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.327595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.558708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.959506] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:18:25 (1778926705) [ 533.118598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.171460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.623636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 533.958637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 541.785213] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:18:33 (1778926713) [ 541.989885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.020747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.085362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.136663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 549.358496] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:18:41 (1778926721) [ 549.470319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.574447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.683942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.754478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.665572] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:18:49 (1778926729) [ 557.839206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.907941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.030698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 565.171203] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:18:57 (1778926737) [ 565.275973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.309256] LNet: 22250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.314882] LNet: 22250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 565.372323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.560561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 573.316228] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:19:05 (1778926745) [ 573.422821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.500218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.608892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 574.168427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 581.221656] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:19:13 (1778926753) [ 581.421032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.550113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.686481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 581.731677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 589.412789] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:19:21 (1778926761) [ 589.577181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.643680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.767776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 589.814757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 597.333357] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:19:29 (1778926769) [ 597.596823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.686420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.743522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 597.794690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 604.675862] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:19:36 (1778926776) [ 604.924241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.976329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.036279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 605.059754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 611.749699] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:19:43 (1778926783) [ 611.904313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.934618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.980307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 612.026377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 612.054624] LNetError: 25790:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 612.090557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 612.136464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 618.028894] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:19:50 (1778926790) [ 618.219862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.276781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.364717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.REMxaDEsRA [ 618.432616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.REMxaDEsRA [ 618.498617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.REMxaDEsRA [ 625.061317] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:19:57 (1778926797) [ 625.267349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.295772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.355686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 625.409190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 625.479267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 632.535861] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:20:04 (1778926804) [ 632.691739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.739994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.870987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 635.563821] LNet: Message forwarding will be enabled in 2s [ 638.623170] LNet: Message forwarding enabled [ 639.062656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 639.073691] LNet: Message forwarding will stop in 2s to 6s [ 643.606363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.169040] LNet: Message forwarding disabled [ 646.177903] LNet: Message forwarding will be enabled in 2s [ 649.317373] LNet: Message forwarding enabled [ 653.202204] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:20:25 (1778926825) [ 653.332524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.147531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.235847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 660.391328] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:20:32 (1778926832) [ 660.599423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.629422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.693558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 660.728731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 660.753830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 660.779166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 660.808090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 660.833540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 660.860521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 660.891195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 660.922640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 660.954540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 660.989704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 661.032977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 661.061442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 661.103979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 661.141922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 661.174373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 661.209513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 661.238484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 661.268922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 661.289153] LNetError: 28859:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 667.200496] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:20:39 (1778926839) [ 667.331749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.379612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.434803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.658417] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:20:45 (1778926845) [ 675.375793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg240-server [ 677.674594] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:20:49 (1778926849) [ 677.890616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.940962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.013386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.029758] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 680.016958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[41-45]@tcp [ 680.042973] LNet: 30576:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 680.106463] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.41@tcp is being used as a gateway but routing feature is not turned on [ 680.124366] LNetError: 30365:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.41@tcp has gone from up to down [ 681.135647] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.41@tcp is being used as a gateway but routing feature is not turned on [ 681.155131] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 682.163526] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.41@tcp is being used as a gateway but routing feature is not turned on [ 682.194065] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 683.169882] LNetError: 30579:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.44@tcp has gone from up to down [ 683.188651] LNetError: 30579:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 684.283692] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 684.290806] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 687.268544] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:20:59 (1778926859) [ 688.364358] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 688.384199] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 689.495890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg240-server [ 691.478428] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:21:03 (1778926863) [ 691.600163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.628703] LNetError: 31506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.653558] LNet: Removed LNI 192.168.202.40@tcp [ 691.736873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.791707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.807789] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 692.315756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 692.330983] LNet: 31628:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 692.337507] LNet: 31628:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 692.350987] LNetError: 31515:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.43@tcp has gone from up to down [ 692.356242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 698.126725] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:21:10 (1778926870) [ 698.344669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.374897] LNet: 32220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.380787] LNet: 32220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 698.393000] LNetError: 32220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.418795] LNet: Removed LNI 192.168.202.40@tcp [ 698.458852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.541816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 698.587773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.623138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.686916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.738726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.767928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.815915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 698.894241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.930755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.007026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.130761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.168531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.226608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 699.315693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.370323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.477033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.565626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.599946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.648860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 699.712781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.739778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.787255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.862887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.893449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.935490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 700.012433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.039594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.103504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.180491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.199467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.262143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 700.315704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.370312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.442829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.497945] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:21:17 (1778926877) [ 706.692304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg240-server [ 708.006523] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:21:20 (1778926880) [ 709.572460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg240-server [ 711.197236] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:21:23 (1778926883) [ 712.091134] Key type .llcrypt unregistered [ 712.093239] Key type ._llcrypt unregistered [ 718.625577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.889698] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.905751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.917514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.457706] Key type ._llcrypt registered [ 721.462559] Key type .llcrypt registered [ 721.666844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.682400] alg: No test for adler32 (adler32-zlib) [ 722.701604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.813115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.850991] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 723.538870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.555183] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.842735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.858894] LNet: Removed LNI 10.1.2.3@tcp [ 725.949644] LNet: 34343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.955552] LNetError: 34343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.988373] LNet: Removed LNI 192.168.202.40@tcp [ 726.525233] Key type .llcrypt unregistered [ 726.527560] Key type ._llcrypt unregistered [ 739.275898] Key type ._llcrypt registered [ 739.279504] Key type .llcrypt registered [ 740.764962] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:21:52 (1778926912) [ 741.380416] Key type .llcrypt unregistered [ 741.382254] Key type ._llcrypt unregistered [ 748.643431] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.753835] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.757365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.762783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.441153] Key type ._llcrypt registered [ 751.444707] Key type .llcrypt registered [ 751.751183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.778081] alg: No test for adler32 (adler32-zlib) [ 752.680500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.736407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.759653] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 753.315697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 753.331323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 755.832740] LNet: 35840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.841216] LNetError: 35840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.853844] LNet: Removed LNI 192.168.202.40@tcp [ 755.862337] LNet: Removed LNI 10.1.2.3@tcp [ 756.563146] Key type .llcrypt unregistered [ 756.568132] Key type ._llcrypt unregistered [ 768.295830] Key type ._llcrypt registered [ 768.297824] Key type .llcrypt registered [ 769.283416] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:22:21 (1778926941) [ 770.015303] Key type .llcrypt unregistered [ 770.017166] Key type ._llcrypt unregistered [ 776.251067] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.346563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.356960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.376432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.204924] Key type ._llcrypt registered [ 779.209774] Key type .llcrypt registered [ 779.496882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.516369] alg: No test for adler32 (adler32-zlib) [ 780.440118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.488717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 780.507807] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 782.330025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 782.352045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.051654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 784.070264] LNet: Removed LNI 10.1.2.3@tcp [ 784.081222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 784.104964] LNet: Removed LNI 10.1.2.31@tcp [ 784.114941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.131040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.682283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 784.695469] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.576619] LNet: 37761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.587359] LNetError: 37761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 785.594410] LNet: Removed LNI 10.1.2.3@tcp [ 785.602809] LNet: Removed LNI 10.1.2.31@tcp [ 786.105608] Key type .llcrypt unregistered [ 786.107888] Key type ._llcrypt unregistered [ 795.797996] Key type ._llcrypt registered [ 795.801494] Key type .llcrypt registered [ 796.900517] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:22:48 (1778926968) [ 797.296253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.312576] alg: No test for adler32 (adler32-zlib) [ 798.209798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.267444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 798.287858] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 799.056198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.086932] LNet: 38714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.094229] LNetError: 38714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.112036] LNet: Removed LNI 192.168.202.40@tcp [ 799.123873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.160341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.40@tcp [ 799.179516] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 799.806613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.40@tcp [ 799.836740] LNet: Removed LNI 192.168.202.40@tcp [ 799.937760] LNet: 38849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.651368] Key type .llcrypt unregistered [ 800.653290] Key type ._llcrypt unregistered [ 810.961154] Key type ._llcrypt registered [ 810.962748] Key type .llcrypt registered [ 812.486448] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:23:04 (1778926984) [ 814.075072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg240-server [ 815.823362] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:23:07 (1778926987) [ 816.361649] Key type .llcrypt unregistered [ 816.365970] Key type ._llcrypt unregistered [ 822.060037] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.247245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.251759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.257959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.814483] Key type ._llcrypt registered [ 824.818638] Key type .llcrypt registered [ 825.079032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.105504] alg: No test for adler32 (adler32-zlib) [ 826.028347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.094473] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 826.909632] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 829.250140] LNet: 40688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.262730] LNetError: 40688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.272539] LNet: Removed LNI 192.168.202.40@tcp1 [ 829.286827] LNet: Removed LNI 10.1.2.3@tcp2 [ 830.081569] Key type .llcrypt unregistered [ 830.086142] Key type ._llcrypt unregistered [ 840.617370] Key type ._llcrypt registered [ 840.618412] Key type .llcrypt registered [ 841.866927] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:23:33 (1778927013) [ 842.240523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.257237] alg: No test for adler32 (adler32-zlib) [ 843.179525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.236273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.264449] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 843.955652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.976282] LNet: 41657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.990757] LNetError: 41657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.009940] LNet: Removed LNI 192.168.202.40@tcp [ 844.030844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.082338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 844.102932] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 850.669951] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:23:42 (1778927022) [ 850.825851] LNet: 42246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.851307] LNetError: 42246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.880211] LNet: Removed LNI 192.168.202.40@tcp [ 851.396724] Key type .llcrypt unregistered [ 851.398784] Key type ._llcrypt unregistered [ 856.990513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.158376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 857.161514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 857.171603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 859.663671] Key type ._llcrypt registered [ 859.665358] Key type .llcrypt registered [ 859.813867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.831780] alg: No test for adler32 (adler32-zlib) [ 860.697835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.741424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.763441] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 861.335547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 861.351475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.850825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.870442] LNet: 42988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.874162] LNetError: 42988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.877753] LNet: Removed LNI 192.168.202.40@tcp [ 862.885634] LNet: Removed LNI 10.1.2.3@tcp [ 862.898511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.935978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 862.953348] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 862.956308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.340612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.370249] LNet: Removed LNI 192.168.202.40@tcp [ 863.382554] LNet: Removed LNI 10.1.2.3@tcp [ 863.414067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.489566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 863.516449] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 863.520697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.074133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 864.096198] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 864.871536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.889420] LNet: 43178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.896165] LNet: 43178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 864.903571] LNetError: 43178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.913390] LNetError: 43178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 864.921166] LNet: Removed LNI 192.168.202.40@tcp [ 864.923920] LNet: Removed LNI 10.1.2.3@tcp [ 864.940916] LNet: Removed LNI 192.168.202.40@tcp2 [ 864.962315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.999752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 865.030578] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 865.038582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.042354] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 871.020828] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:24:03 (1778927043) [ 871.268549] LNet: 43776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.274719] LNetError: 43776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.278608] LNetError: 43776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 871.283406] LNet: Removed LNI 192.168.202.40@tcp [ 871.284812] LNet: Removed LNI 10.1.2.3@tcp [ 871.296921] LNet: Removed LNI 192.168.202.40@tcp2 [ 871.764798] Key type .llcrypt unregistered [ 871.768172] Key type ._llcrypt unregistered [ 877.235589] Key type ._llcrypt registered [ 877.238555] Key type .llcrypt registered [ 877.452302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.475846] alg: No test for adler32 (adler32-zlib) [ 878.438165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.488616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.515175] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 879.365799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.390835] LNet: 44171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.403075] LNetError: 44171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.422531] LNet: Removed LNI 192.168.202.40@tcp [ 879.457571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.543563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 879.570796] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 880.122533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.145759] LNet: 44201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.154274] LNetError: 44201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.162892] LNet: Removed LNI 192.168.202.40@tcp [ 880.188167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.260381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 880.276604] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 880.807837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.839522] LNet: Removed LNI 192.168.202.40@tcp [ 880.902799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.032049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 881.086705] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 881.656768] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.832183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.840019] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.845151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 884.171140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.199258] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.208437] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 884.216501] LNetError: 44309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.222297] LNetError: 44309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.244878] LNet: Removed LNI 192.168.202.40@tcp [ 884.265908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.320876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 884.346430] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 889.557932] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:24:21 (1778927061) [ 889.683548] LNet: 44903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.688547] LNetError: 44903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.698933] LNet: Removed LNI 192.168.202.40@tcp [ 890.152813] Key type .llcrypt unregistered [ 890.154663] Key type ._llcrypt unregistered [ 894.662962] Key type ._llcrypt registered [ 894.664450] Key type .llcrypt registered [ 894.792413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.802408] alg: No test for adler32 (adler32-zlib) [ 895.668620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.714693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.733156] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 896.364806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.374844] LNet: 45305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.377757] LNetError: 45305:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.390930] LNet: Removed LNI 192.168.202.40@tcp [ 896.406569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.464695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 896.476393] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 901.253942] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:24:33 (1778927073) [ 901.332887] LNet: 45890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.338686] LNetError: 45890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.381864] LNet: Removed LNI 192.168.202.40@tcp [ 902.025126] Key type .llcrypt unregistered [ 902.027334] Key type ._llcrypt unregistered [ 906.921796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.980096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.984922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.988644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 908.467648] Key type ._llcrypt registered [ 908.469615] Key type .llcrypt registered [ 908.620795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.630662] alg: No test for adler32 (adler32-zlib) [ 909.530466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.587172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.603395] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 910.358788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.382296] LNet: 46335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.386609] LNetError: 46335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.407445] LNet: Removed LNI 192.168.202.40@tcp [ 910.430927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.533972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 910.554554] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 916.178609] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:24:48 (1778927088) [ 916.270923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.299311] LNet: 46929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.307661] LNetError: 46929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.320089] LNet: Removed LNI 192.168.202.40@tcp [ 916.345800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.437712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 920.930164] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:24:53 (1778927093) [ 921.048854] LNet: 47515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.487459] Key type .llcrypt unregistered [ 921.490238] Key type ._llcrypt unregistered [ 925.335301] Key type ._llcrypt registered [ 925.344173] Key type .llcrypt registered [ 925.561409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.574844] alg: No test for adler32 (adler32-zlib) [ 926.479586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.528334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.548913] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 927.342373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.380885] LNet: 47912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.390853] LNetError: 47912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.403463] LNet: Removed LNI 192.168.202.40@tcp [ 927.418567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.470903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 927.503084] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 932.492796] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:25:04 (1778927104) [ 932.632045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.682929] LNet: 48505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.690962] LNetError: 48505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.701828] LNet: Removed LNI 192.168.202.40@tcp [ 932.719814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.775813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 937.430796] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:25:09 (1778927109) [ 938.532398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 939.919972] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:25:12 (1778927112) [ 941.073476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 942.549102] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:25:14 (1778927114) [ 943.694744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 944.757253] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:25:17 (1778927117) [ 945.774975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 947.111575] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:25:19 (1778927119) [ 948.370855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 949.617334] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:25:21 (1778927121) [ 949.769758] LNet: 50554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.197588] Key type .llcrypt unregistered [ 950.200319] Key type ._llcrypt unregistered [ 953.719228] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.854276] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.874459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.881672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.258394] Key type ._llcrypt registered [ 956.260870] Key type .llcrypt registered [ 956.391551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.405777] alg: No test for adler32 (adler32-zlib) [ 957.260444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.308312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.321512] LNet: Added LNI 192.168.202.40@tcp [8/512/0/180] [ 957.326035] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 957.327594] LNet: 50876:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.388425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.398985] LNet: 50895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.402664] LNetError: 50895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.405991] LNet: Removed LNI 192.168.202.40@tcp [ 957.408180] LNetError: 50895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.420148] LNet: Removed LNI 10.1.2.3@tcp [ 957.430261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.481314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.494586] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 957.537819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.553868] LNet: 50919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.558184] LNetError: 50919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.572823] LNet: Removed LNI 192.168.202.40@tcp [ 957.589135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.630806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.647530] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 957.651248] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.652793] LNet: 50927:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.707620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.722487] LNet: 50945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.726603] LNetError: 50945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.732237] LNet: Removed LNI 192.168.202.40@tcp [ 957.733714] LNetError: 50945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.744745] LNet: Removed LNI 10.1.2.3@tcp [ 957.756752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.809374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.827523] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 957.830741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.834211] LNet: 50953:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.990502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.008512] LNet: 50979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.013344] LNetError: 50979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.016855] LNet: Removed LNI 192.168.202.40@tcp [ 958.018780] LNetError: 50979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.027962] LNet: Removed LNI 10.1.2.3@tcp [ 958.041848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.094340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.114410] LNet: Added LNI 192.168.202.40@tcp [8/256/0/182] [ 958.117862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 958.188163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.203717] LNet: 51007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.207964] LNetError: 51007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.211655] LNet: Removed LNI 192.168.202.40@tcp [ 958.214663] LNetError: 51007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.223762] LNet: Removed LNI 10.1.2.3@tcp [ 958.237257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.285297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.300409] LNet: Added LNI 192.168.202.40@tcp [8/256/0/182] [ 958.304376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 958.306566] LNet: 51015:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 958.359498] LNet: Removed LNI 192.168.202.40@tcp [ 958.368898] LNet: Removed LNI 10.1.2.3@tcp [ 958.774527] Key type .llcrypt unregistered [ 958.776955] Key type ._llcrypt unregistered [ 966.431649] Key type ._llcrypt registered [ 966.437179] Key type .llcrypt registered [ 967.396631] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:25:39 (1778927139) [ 967.743219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.752568] alg: No test for adler32 (adler32-zlib) [ 968.630853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.666226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.683143] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 969.276940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.40 [ 969.287643] LNet: 51991:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 969.291923] LNet: Removed LNI 192.168.202.40@tcp [ 973.839155] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:25:46 (1778927146) [ 973.928827] LNet: 52552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.933549] LNetError: 52552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.951082] LNet: Removed LNI 192.168.202.40@tcp [ 974.397667] Key type .llcrypt unregistered [ 974.400172] Key type ._llcrypt unregistered [ 979.437372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.599548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 982.244327] Key type ._llcrypt registered [ 982.247972] Key type .llcrypt registered [ 982.377466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.388495] alg: No test for adler32 (adler32-zlib) [ 983.481848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.005730] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:26:00 (1778927160) [ 988.119661] LNet: 53469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.129921] LNetError: 53469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 988.144236] LNet: Removed LNI 10.1.2.3@tcp [ 988.646592] Key type .llcrypt unregistered [ 988.648787] Key type ._llcrypt unregistered [ 992.454628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 992.626291] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 992.630978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 992.633376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.056130] Key type ._llcrypt registered [ 995.057161] Key type .llcrypt registered [ 995.162595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.178352] alg: No test for adler32 (adler32-zlib) [ 996.088427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.315944] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:26:12 (1778927172) [ 1001.251639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1002.278833] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:26:14 (1778927174) [ 1002.464976] LNet: 54673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.476676] LNetError: 54673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.494215] LNet: Removed LNI 10.1.2.3@tcp [ 1002.864629] Key type .llcrypt unregistered [ 1002.865620] Key type ._llcrypt unregistered [ 1007.321201] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1007.497894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1007.501315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.506357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1010.037146] Key type ._llcrypt registered [ 1010.042299] Key type .llcrypt registered [ 1010.191439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.204314] alg: No test for adler32 (adler32-zlib) [ 1011.066677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.120293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.926324] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:26:29 (1778927189) [ 1017.176793] LNet: 55901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.179979] LNetError: 55901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1017.189741] LNet: Removed LNI 10.1.2.3@tcp [ 1017.521037] Key type .llcrypt unregistered [ 1017.522855] Key type ._llcrypt unregistered [ 1021.484982] Key type ._llcrypt registered [ 1021.486436] Key type .llcrypt registered [ 1021.635528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.655083] alg: No test for adler32 (adler32-zlib) [ 1022.546725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1022.564957] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1027.899997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1028.495101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1028.523508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1028.547404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1028.564632] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1028.704527] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1028.795982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.818755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.841276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.887974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.922373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.963972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.009100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.066867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.091735] LNet: 56338:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1029.120801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.165478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.195573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.227715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.257822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.286166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.303878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.729967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1029.749169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1029.780527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1029.797857] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1030.034829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.074594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.096850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.120519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.156276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.189753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.218917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.248846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.262201] LNet: 56447:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.276155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.296947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.321194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.346887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.376528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.398977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.428527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.862870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1030.896630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1030.923126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1030.939736] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1031.158796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.181740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1031.203129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.227221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.259752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.293724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.318723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.342138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.352735] LNet: 56556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1031.360585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.377964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.403709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.430783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.461691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.493588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.522847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.972994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1032.002711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1032.032808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1032.248559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.274946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1032.293238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.309398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.348170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.423502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.498227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.547409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.592308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.636162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.665456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.692976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.744436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.783950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.827199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.218655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1033.242687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1033.280307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1033.296536] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1033.299833] LNet: Skipped 1 previous similar message [ 1033.530607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.565890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.603488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.628880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.644809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.665041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.690543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.712888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.727496] LNet: 56775:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.732413] LNet: 56775:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1033.743173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.767927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.799793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.818442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.835547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.855390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.874796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.312960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1034.349896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1034.403293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1034.618231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.642213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.671178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.700762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.728543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.777987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.805927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.831781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.852478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.871615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.901929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.923751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.955848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.990456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.017966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1035.267470] LNet: 56925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.276980] LNetError: 56925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.296957] LNet: Removed LNI 192.168.202.40@tcp [ 1035.590570] Key type .llcrypt unregistered [ 1035.593388] Key type ._llcrypt unregistered [ 1047.537566] Key type ._llcrypt registered [ 1047.539883] Key type .llcrypt registered [ 1048.592232] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:27:00 (1778927220) [ 1049.248598] Key type .llcrypt unregistered [ 1049.253934] Key type ._llcrypt unregistered [ 1054.327653] Key type ._llcrypt registered [ 1054.331777] Key type .llcrypt registered [ 1054.506871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.515504] alg: No test for adler32 (adler32-zlib) [ 1055.411635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1055.427561] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1061.116292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1068.820394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.868805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1068.889959] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1069.952728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1069.979324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 1070.000780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1070.041693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 1070.085167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1070.126507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 1070.169962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1070.199160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 1070.237071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1070.249660] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1070.256373] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1070.384624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.401788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1070.423431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.453935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.482135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.519896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.549360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.579579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.596294] LNet: 58392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.608573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.631976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.650859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.668482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.690655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.714585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.735656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1071.117345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1071.143119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 1071.172260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1071.206441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 1071.250793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1071.282182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 1071.309583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1071.339795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 1071.365670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1071.386246] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1071.389948] LNet: Skipped 2 previous similar messages [ 1071.601487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.631673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1071.661643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.690879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.717856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.756689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.814175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.846794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.855950] LNet: 58513:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1071.869700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.893791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.917111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.946533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.967985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.002745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.030149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1072.435646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1072.479275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 1072.519454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1072.548402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 1072.581518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1072.610204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 1072.658689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1072.682336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 1072.699714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1072.709726] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1072.712031] LNet: Skipped 2 previous similar messages [ 1072.870389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.900167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1072.928101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.951543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.972573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.004213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.027389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.051740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.072471] LNet: 58635:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1073.088824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.118061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.148439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.171534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.196715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.224504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.250396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1073.647921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1073.686605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 1073.716275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1073.748730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 1073.778469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1073.808926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 1073.836209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1073.862838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 1073.893799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1074.078753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.109727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1074.133607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.160524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.181522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.206885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.231368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.253299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.273542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.300410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.324781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.346477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.365548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.389559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.417831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1074.818561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1074.844832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 1074.871472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1074.902656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 1074.935204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1074.962787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 1074.993191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1075.020355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 1075.052641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1075.068487] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1075.072795] LNet: Skipped 5 previous similar messages [ 1075.273647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.298861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.325478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.348861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.374591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.400518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.425631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.450065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.465969] LNet: 58877:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.470324] LNet: 58877:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1075.494328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.529833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.569873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.616174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.678215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.727762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.756533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.111499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1076.140138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 1076.167325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1076.213723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 1076.254247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1076.293606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 1076.321422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1076.346435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 1076.403135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1076.615647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.647644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1076.680820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.706447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.731339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.753293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.777503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.825390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.874200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.939578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.966785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.011570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.040984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.094246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.130397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.353706] LNet: 59074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.359940] LNetError: 59074:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.370066] LNet: Removed LNI 192.168.202.40@tcp [ 1081.390100] LNet: Removed LNI 192.168.202.40@tcp1 [ 1081.771636] Key type .llcrypt unregistered [ 1081.774260] Key type ._llcrypt unregistered [ 1094.440637] Key type ._llcrypt registered [ 1094.443219] Key type .llcrypt registered [ 1095.505038] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:27:47 (1778927267) [ 1096.219598] Key type .llcrypt unregistered [ 1096.222673] Key type ._llcrypt unregistered [ 1100.470186] Key type ._llcrypt registered [ 1100.471929] Key type .llcrypt registered [ 1100.596388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.607367] alg: No test for adler32 (adler32-zlib) [ 1101.472982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.493333] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1106.337128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1106.910690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1106.932957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1106.964615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1106.979196] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1107.126358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.149150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.180240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.217207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.261470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.294373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.316137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.334791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.346243] LNet: 60322:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.364564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.386557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.403838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.421907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.441672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.467447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.508159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.925192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1107.947785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1107.980224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1108.002541] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1108.153348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.168789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1108.193047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.217291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.237985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.256568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.273915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.298775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.312151] LNet: 60432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.325908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.352953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.375354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.400067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.428314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.457134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.477555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1108.866684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1108.905594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1108.935303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1109.178686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.198436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.215877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.230097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.244511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.258358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.275996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.289591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.304210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.319545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.334237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.347114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.366414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.395208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.425389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.641112] LNet: 60583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.644285] LNetError: 60583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.653647] LNet: Removed LNI 192.168.202.40@tcp [ 1109.947491] Key type .llcrypt unregistered [ 1109.948525] Key type ._llcrypt unregistered [ 1122.087799] Key type ._llcrypt registered [ 1122.091992] Key type .llcrypt registered [ 1123.060260] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:28:15 (1778927295) [ 1123.764262] Key type .llcrypt unregistered [ 1123.766062] Key type ._llcrypt unregistered [ 1128.306219] Key type ._llcrypt registered [ 1128.308918] Key type .llcrypt registered [ 1128.476840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.489969] alg: No test for adler32 (adler32-zlib) [ 1129.382718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1129.412752] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1135.030485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1142.839174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.886371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1142.910934] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1143.851129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1143.882507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 1143.914850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1143.941952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 1143.975395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1144.000621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 1144.024462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1144.052119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 1144.084747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1144.108597] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1144.115786] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1144.300785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.321388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1144.344952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.368848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.398336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.420814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.441114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.466480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.480068] LNet: 62056:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.499405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.523117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.554332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.588332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.623933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.655646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.704132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1144.731755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.160332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1145.187936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 1145.212119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1145.241892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 1145.273391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1145.302192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 1145.326500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1145.348966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 1145.381113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1145.395948] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1145.398785] LNet: Skipped 2 previous similar messages [ 1145.576679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.608566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1145.632078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.666228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.692900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.731491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.762058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.803863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.822173] LNet: 62179:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.842307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.877735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.902094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.921520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.950950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.985929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.015917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.045337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.468914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1146.507570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 1146.556141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1146.590347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 1146.633817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1146.668760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 1146.710439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1146.763406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 1146.794189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1146.811894] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1147.030102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.051905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1147.076041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.096467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.119841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.147281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.175643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.199840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.215938] LNet: 62302:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.231506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.260979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.297669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.320459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.344870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.367868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.402925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1147.436484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.458572] LNet: 62380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.464294] LNetError: 62380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.479233] LNet: Removed LNI 192.168.202.40@tcp [ 1152.483688] LNetError: 62380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.508334] LNet: Removed LNI 192.168.202.40@tcp1 [ 1152.937989] Key type .llcrypt unregistered [ 1152.939211] Key type ._llcrypt unregistered [ 1163.855925] Key type ._llcrypt registered [ 1163.857734] Key type .llcrypt registered [ 1164.811340] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:28:57 (1778927337) [ 1165.869989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1167.030904] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:28:59 (1778927339) [ 1167.562256] Key type .llcrypt unregistered [ 1167.568848] Key type ._llcrypt unregistered [ 1171.795638] Key type ._llcrypt registered [ 1171.798420] Key type .llcrypt registered [ 1171.948211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.960905] alg: No test for adler32 (adler32-zlib) [ 1172.838923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.868418] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1177.979740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1178.523395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1178.572351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1178.631991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1178.644703] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1178.818765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.841476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1178.863298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.881476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.910586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.930449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.950858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.974487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.984986] LNet: 63922:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.001522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.031678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.041342] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1179.070227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.114971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.162425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.192041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.428058] LNet: 63961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.434202] LNetError: 63961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.462374] LNet: Removed LNI 192.168.202.40@tcp [ 1179.758585] Key type .llcrypt unregistered [ 1179.760107] Key type ._llcrypt unregistered [ 1192.775210] Key type ._llcrypt registered [ 1192.777335] Key type .llcrypt registered [ 1192.976425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1193.003138] alg: No test for adler32 (adler32-zlib) [ 1193.923522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1193.947038] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1200.195405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1207.676299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.717520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.736422] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1208.612708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1208.642234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 1208.690519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1208.748860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 1208.795583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1208.831683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 1208.854771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1208.900431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 1208.952234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1208.985052] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1209.231504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.264950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1209.295179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.314427] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1209.319074] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1209.329135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.359523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.390962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.418823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.441159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.453867] LNet: 64838:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.469724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.493176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.512406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.534203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.556284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.577482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.681387] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1213.694892] LNet: Skipped 2 previous similar messages [ 1214.018102] LNet: 64912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.022910] LNetError: 64912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.029370] LNet: Removed LNI 192.168.202.40@tcp [ 1214.040155] LNet: Removed LNI 192.168.202.40@tcp1 [ 1214.384600] Key type .llcrypt unregistered [ 1214.388325] Key type ._llcrypt unregistered [ 1227.544451] Key type ._llcrypt registered [ 1227.546410] Key type .llcrypt registered [ 1228.679418] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:30:00 (1778927400) [ 1228.907090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.920658] alg: No test for adler32 (adler32-zlib) [ 1229.807967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.877800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1229.900573] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1230.487930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1230.505234] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1231.141362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1231.197824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1231.232753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1231.262370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1231.299992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1231.338824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1231.362342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1231.376839] LNetError: 66010:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1231.904285] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1231.905789] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1233.952086] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1234.749126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1237.023567] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1238.080402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1238.093094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.133225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.146650] LNet: 66117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.155445] LNetError: 66117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.199147] LNet: Removed LNI 192.168.202.40@tcp [ 1239.202828] LNetError: 66117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.225113] LNet: Removed LNI 192.168.202.40@tcp1 [ 1239.258323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.317589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1239.334786] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1239.743699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1239.762801] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1240.319623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1240.346994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.395785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1240.423175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1240.448955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1240.471566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1240.498697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1240.522577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1240.536903] LNetError: 66390:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1241.375513] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1243.914772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1247.519407] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1247.522171] LNet: Skipped 1 previous similar message [ 1252.746188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1252.770373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.792880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.253269] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:30:29 (1778927429) [ 1257.401943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.422211] LNet: 67116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.426795] LNetError: 67116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.431076] LNet: Removed LNI 192.168.202.40@tcp [ 1257.451576] LNet: Removed LNI 192.168.202.40@tcp1 [ 1257.467581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.511954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1257.531035] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1258.030830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.047777] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1258.651172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1258.705437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1258.732505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1258.768621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1258.793600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1258.822285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1258.842389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 1258.913380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1259.551407] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.40@tcp [ 1259.555017] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1259.558045] LNet: Skipped 1 previous similar message [ 1273.975696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1274.096357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1274.124927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.149251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 500 [ 1274.209331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1280.681537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1280.722738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1280.737642] LNet: 67607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.746433] LNetError: 67607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.751207] LNetError: 67607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1280.756575] LNet: Removed LNI 192.168.202.40@tcp [ 1280.768621] LNet: Removed LNI 192.168.202.40@tcp1 [ 1280.782520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.824507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1280.841243] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1281.334513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1281.349231] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1281.844831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1281.887529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.953814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1281.981829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1282.011645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1282.037148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1282.063499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1282.120797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 1283.875133] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1283.885806] LNet: Skipped 3 previous similar messages [ 1284.449481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1294.440848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1294.496412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.541747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.313740] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:31:12 (1778927472) [ 1300.714461] LNet: 68663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.718803] LNetError: 68663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.724370] LNetError: 68663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1300.728192] LNet: Removed LNI 192.168.202.40@tcp [ 1300.734707] LNet: Removed LNI 192.168.202.40@tcp1 [ 1301.134698] Key type .llcrypt unregistered [ 1301.137567] Key type ._llcrypt unregistered [ 1305.498542] Key type ._llcrypt registered [ 1305.505567] Key type .llcrypt registered [ 1305.702860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.719436] alg: No test for adler32 (adler32-zlib) [ 1306.589908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1306.609100] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1313.280517] Lustre: DEBUG MARKER: Initial discovery [ 1313.322420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1315.126629] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1315.323077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1315.361655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1315.398151] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1315.417102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.441561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -l 3 [ 1315.475938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -e local_error [ 1315.508596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m PUT -l 6 [ 1315.871977] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1316.756673] Lustre: DEBUG MARKER: Force oleg240-client.virtnet to discover 192.168.202.140@tcp (in background) [ 1316.871471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1318.132653] Lustre: DEBUG MARKER: Force oleg240-server to discover 192.168.202.40@tcp [ 1319.871504] Lustre: DEBUG MARKER: Wait for 69418 [ 1319.969202] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1321.082293] Lustre: DEBUG MARKER: Finished wait on 69418 [ 1321.344475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1321.369808] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1321.377309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1321.615944] LNet: 69874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.625896] LNetError: 69874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.647398] LNet: Removed LNI 192.168.202.40@tcp [ 1321.928889] Key type .llcrypt unregistered [ 1321.931081] Key type ._llcrypt unregistered [ 1333.557698] Key type ._llcrypt registered [ 1333.561291] Key type .llcrypt registered [ 1334.515656] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:31:46 (1778927506) [ 1334.874261] Key type .llcrypt unregistered [ 1334.876659] Key type ._llcrypt unregistered [ 1338.545720] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1338.649018] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1338.651817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1338.654842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1341.065835] Key type ._llcrypt registered [ 1341.067433] Key type .llcrypt registered [ 1341.174258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1341.182697] alg: No test for adler32 (adler32-zlib) [ 1342.084849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.122838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1342.143119] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1342.709461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1342.729530] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1349.113680] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:32:01 (1778927521) [ 1349.285480] LNet: 71946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.289362] LNetError: 71946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.294533] LNet: Removed LNI 192.168.202.40@tcp [ 1349.312219] LNet: Removed LNI 10.1.2.3@tcp [ 1349.773418] Key type .llcrypt unregistered [ 1349.775492] Key type ._llcrypt unregistered [ 1354.372856] Key type ._llcrypt registered [ 1354.373975] Key type .llcrypt registered [ 1354.475467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.482182] alg: No test for adler32 (adler32-zlib) [ 1355.362176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.396214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1355.425042] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1355.886419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1355.903051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1356.642723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.40@tcp [ 1357.732291] LNetError: Unexpected error -22 connecting to 192.168.202.40@tcp at host 192.168.202.40:988 [ 1360.997932] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:32:13 (1778927533) [ 1361.108830] LNet: 73076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.114350] LNetError: 73076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.118871] LNet: Removed LNI 192.168.202.40@tcp [ 1361.127900] LNet: Removed LNI 10.1.2.3@tcp [ 1361.699629] Key type .llcrypt unregistered [ 1361.702863] Key type ._llcrypt unregistered [ 1366.327834] Key type ._llcrypt registered [ 1366.329894] Key type .llcrypt registered [ 1366.512659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1366.527336] alg: No test for adler32 (adler32-zlib) [ 1367.437802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.519079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.563711] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1368.085345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1368.096916] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1368.728217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.40@tcp1 --nid 192.168.202.40@tcp2 [ 1371.611996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp2 [ 1371.633107] LNetError: 73916:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.40@tcp1 to 192.168.202.40@tcp2 - Route table may be misconfigured [ 1371.657567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp2 192.168.202.40@tcp1 [ 1375.974717] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:32:28 (1778927548) [ 1376.048926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1376.071546] LNet: 74485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.079371] LNetError: 74485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.119426] LNet: Removed LNI 192.168.202.40@tcp1 [ 1377.124089] LNetError: 74485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.145691] LNet: Removed LNI 192.168.202.40@tcp2 [ 1377.173995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.205910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1377.219090] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1377.703524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1377.719669] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1378.321377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1378.344996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.40@tcp [ 1378.367836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp1 [ 1378.395740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp -e network_timeout [ 1378.433449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -e network_timeout [ 1378.459373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -e network_timeout [ 1378.483411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -e network_timeout [ 1378.505485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1378.518830] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1378.528962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1378.582209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.648853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1379.232698] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1382.598541] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:32:35 (1778927555) [ 1382.677421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.703491] LNet: 75346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.712445] LNetError: 75346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.718762] LNet: Removed LNI 192.168.202.40@tcp [ 1383.787149] LNet: Removed LNI 192.168.202.40@tcp1 [ 1383.811620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.859190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1384.006207] LNet: 75387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.324294] Key type .llcrypt unregistered [ 1384.325916] Key type ._llcrypt unregistered [ 1391.695784] Key type ._llcrypt registered [ 1391.697983] Key type .llcrypt registered [ 1392.598304] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:32:44 (1778927564) [ 1392.930232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1392.941308] alg: No test for adler32 (adler32-zlib) [ 1393.814730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.863444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1396.127880] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:32:48 (1778927568) [ 1396.179224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.194210] LNet: 76493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.210395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.264401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.284785] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1396.774591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1396.795747] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1397.387826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1397.403776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp1 [ 1397.423958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp1 [ 1401.801303] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:32:54 (1778927574) [ 1403.010537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg240-server [ 1404.315111] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:32:56 (1778927576) [ 1405.568952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg240-server [ 1406.465065] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:32:59 (1778927579) [ 1407.537334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg240-server [ 1408.492287] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:33:01 (1778927581) [ 1409.593349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg240-server [ 1410.614082] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:33:03 (1778927583) [ 1411.667709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg240-server [ 1412.619788] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:33:05 (1778927585) [ 1413.817290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg240-server [ 1414.908746] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:33:07 (1778927587) [ 1415.951978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg240-server [ 1417.074198] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:33:09 (1778927589) [ 1418.260421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg240-server [ 1419.471379] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:33:11 (1778927591) [ 1420.618495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg240-server [ 1421.951806] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:33:14 (1778927594) [ 1422.117064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.134691] LNet: 80378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.140617] LNetError: 80378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.152274] LNet: Removed LNI 192.168.202.40@tcp [ 1423.212393] LNet: Removed LNI 192.168.202.40@tcp1 [ 1423.223972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.272095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.286785] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1423.756958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1423.803833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1423.931958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.957153] LNet: 80525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.967913] LNetError: 80525:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.974076] LNetError: 80525:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.998176] LNet: Removed LNI 192.168.202.40@tcp [ 1424.025149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.068405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1424.096400] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1424.759985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1424.915499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.962958] LNet: Removed LNI 192.168.202.40@tcp [ 1425.974207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.015277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1426.048703] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1426.572218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1426.633316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1426.762411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.775434] LNet: 80818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.778040] LNet: 80818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1426.781361] LNetError: 80818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.786746] LNetError: 80818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1426.798132] LNet: Removed LNI 192.168.202.40@tcp [ 1426.814465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.850823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1426.864450] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1427.398163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1427.535469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.582353] LNet: Removed LNI 192.168.202.40@tcp [ 1428.612989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.677611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.708296] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1429.168215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1429.219467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1429.355645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.370298] LNetError: 81110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.374364] LNetError: 81110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1429.389845] LNet: Removed LNI 192.168.202.40@tcp [ 1429.405437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.432903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1429.454536] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1429.958393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1431.161916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.176079] LNet: 81264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.180436] LNet: 81264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1432.232717] LNet: Removed LNI 192.168.202.40@tcp [ 1432.263947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.322585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.337806] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1432.806375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1432.853227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1432.977857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.022801] LNet: Removed LNI 192.168.202.40@tcp [ 1434.037578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.084716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1434.105609] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1434.584964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1435.829744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.849120] LNetError: 81566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.854277] LNetError: 81566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1435.861467] LNet: Removed LNI 192.168.202.40@tcp [ 1435.876886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.927378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.953726] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1436.420858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1436.472448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1436.618426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.670268] LNet: Removed LNI 192.168.202.40@tcp [ 1437.694318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.731934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1437.750250] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1438.314584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1438.498537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.526902] LNet: Removed LNI 192.168.202.40@tcp [ 1439.557383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.609887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.627154] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1440.146954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1440.222671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1440.372090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.392442] LNet: 82006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.396713] LNet: 82006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1441.452974] LNet: Removed LNI 192.168.202.40@tcp [ 1441.466673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.523412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1441.539190] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1442.017423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1443.336427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.399964] LNet: Removed LNI 192.168.202.40@tcp [ 1444.432518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.484786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.512828] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1444.995653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1445.069186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.090977] LNetError: 82292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.096704] LNetError: 82292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1445.110316] LNet: Removed LNI 192.168.202.40@tcp [ 1445.121493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.173795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.198046] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1445.714223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1449.875760] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:33:42 (1778927622) [ 1449.955694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.984925] LNet: Removed LNI 192.168.202.40@tcp [ 1450.001794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.045360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1450.057060] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1450.736582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.763836] LNet: Removed LNI 192.168.202.40@tcp231 [ 1450.776717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.819720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1450.835161] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1450.838274] LNet: 83142:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.107445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1451.141672] LNet: Removed LNI 192.168.202.40@tcp231 [ 1451.153332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1451.182339] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1452.076173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.098931] LNet: Removed LNI 192.168.202.40@tcp231 [ 1452.117344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.174716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1452.204745] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 1452.209995] LNet: 83294:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1452.601603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1452.638958] LNet: Removed LNI 192.168.202.40@tcp231 [ 1452.665762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1452.686439] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 1453.555459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.583960] LNet: Removed LNI 192.168.202.40@tcp231 [ 1453.613094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.673328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1453.693636] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 1453.697506] LNet: 83446:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1453.918472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1453.949128] LNet: Removed LNI 192.168.202.40@tcp231 [ 1453.967959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1453.987778] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 1454.711392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.732921] LNet: Removed LNI 192.168.202.40@tcp231 [ 1454.759385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.829928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1454.846290] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1455.084220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1455.110940] LNet: Removed LNI 192.168.202.40@tcp231 [ 1455.123350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1455.145618] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1459.832546] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:33:52 (1778927632) [ 1459.910675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.934364] LNet: 84306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.938065] LNet: 84306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1459.950910] LNet: Removed LNI 192.168.202.40@tcp231 [ 1459.962631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.005644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1460.024693] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1460.501919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1460.531726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1460.554795] LNet: Removed LNI 192.168.202.40@tcp232 [ 1460.571725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1460.589704] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1461.043514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1461.067598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1461.103086] LNet: Removed LNI 192.168.202.40@tcp232 [ 1461.120792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1461.141861] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1461.591893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1461.618410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1461.645399] LNet: Removed LNI 192.168.202.40@tcp232 [ 1461.658094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1461.681047] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1462.175627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1462.211318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1462.240757] LNet: Removed LNI 192.168.202.40@tcp232 [ 1462.258590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1466.099425] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:33:58 (1778927638) [ 1466.921460] Key type .llcrypt unregistered [ 1466.923602] Key type ._llcrypt unregistered [ 1470.441974] Key type ._llcrypt registered [ 1470.444160] Key type .llcrypt registered [ 1470.582881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.594858] alg: No test for adler32 (adler32-zlib) [ 1471.448180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1471.465917] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1476.655886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1481.386444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.421963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.437800] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1482.339409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 1482.366771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 1482.386768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 1482.401642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 1482.416570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 1482.430642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 1482.449344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 1482.476773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 1482.499359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1482.512287] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1482.515434] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1482.661106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1482.678297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1482.694991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1482.713111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1482.732336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1482.755986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1482.777665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1482.798298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1482.808687] LNet: 86096:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1482.823841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1482.850764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1482.869574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1482.888593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1482.905194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1482.923809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1483.272567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 1483.310804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 1483.335861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 1483.355604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 1483.384071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 1483.407306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 1483.433859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 1483.458561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 1483.482970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1483.498466] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1483.652952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1483.680706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1483.702273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1483.723659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1483.746768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1483.764495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1483.784225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1483.800620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1483.810450] LNet: 86216:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1483.823402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1483.846782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1483.864650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.891699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.918316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1483.943699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1484.299379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 1484.319273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 1484.342525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 1484.365897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 1484.384722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 1484.411947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 1484.435580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 1484.458790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 1484.500389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1484.517924] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1484.521151] LNet: Skipped 1 previous similar message [ 1484.676071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1484.692333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1484.709833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1484.729971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1484.758589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1484.784839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1484.805994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1484.824505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1484.843175] LNet: 86337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1484.853920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1484.873526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1484.895837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1484.915485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1484.944311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1484.964381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1485.324419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 1485.364642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 1485.395403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 1485.421223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 1485.447864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 1485.480244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 1485.510825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 1485.543961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 1485.579843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1485.792275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.813851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1485.839281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.863457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.878089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.900972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.919688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.940756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.961834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.983292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.005305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.022695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.039900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.062939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.401821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 1486.428481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 1486.457050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 1486.491915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 1486.523788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 1486.567938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 1486.594411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 1486.622191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 1486.644737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1486.660611] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1486.664244] LNet: Skipped 1 previous similar message [ 1486.808492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.824564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1486.841786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.864402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.882599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.905160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.934677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.973075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.990703] LNet: 86577:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.994911] LNet: 86577:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1487.013282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1487.033930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1487.051687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1487.068243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1487.097655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1487.129839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1487.495458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 1487.524266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 1487.556129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 1487.580570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 1487.604666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 1487.625457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 1487.642831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 1487.668780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 1487.706195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1487.729429] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1487.897083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.932073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1487.956933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.986312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1488.013970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1488.047666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1488.091247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1488.124628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1488.155568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1488.197700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1488.235209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.252942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.277405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.306214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1491.433961] LNet: 86771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.437896] LNetError: 86771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.447240] LNet: Removed LNI 192.168.202.40@tcp [ 1492.449399] LNetError: 86771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.462040] LNet: Removed LNI 192.168.202.40@tcp1 [ 1492.678496] Key type .llcrypt unregistered [ 1492.679703] Key type ._llcrypt unregistered [ 1502.211151] Key type ._llcrypt registered [ 1502.212723] Key type .llcrypt registered [ 1503.045317] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:34:35 (1778927675) [ 1503.308976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.316639] alg: No test for adler32 (adler32-zlib) [ 1504.172598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.212913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.230528] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1504.868767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1504.888621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1504.899636] LNetError: 87748:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1504.950453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.40@tcp [ 1505.051226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1508.953804] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:34:41 (1778927681) [ 1509.087654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.100902] LNet: 88337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.106111] LNetError: 88337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.118886] LNet: Removed LNI 192.168.202.40@tcp [ 1509.131249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.159836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.174522] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1509.644901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1509.661657] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1510.343780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.40@tcp [ 1510.361521] LNetError: 88604:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1511.199776] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1511.203071] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1511.528347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.40@tcp [ 1517.602556] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:34:50 (1778927690) [ 1518.779395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg240-server [ 1519.921166] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:34:52 (1778927692) [ 1521.177573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg240-server [ 1522.593552] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:34:54 (1778927694) [ 1523.854432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg240-server [ 1524.974652] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:34:57 (1778927697) [ 1525.085386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.114135] LNet: 90231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.117327] LNetError: 90231:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.129277] LNet: Removed LNI 192.168.202.40@tcp [ 1525.156372] LNet: Removed LNI 192.168.202.40@tcp1 [ 1525.174508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.210823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.225410] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1525.772736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1525.836335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1525.874530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1525.933957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1525.975693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1526.000586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1530.008342] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:35:02 (1778927702) [ 1530.110928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.134407] LNet: 90956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.166634] LNetError: 90956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.173603] LNetError: 90956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1530.193832] LNet: Removed LNI 192.168.202.40@tcp [ 1530.229851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.271404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.287552] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1534.951140] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:35:07 (1778927707) [ 1535.005534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.018232] LNet: 91656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.022966] LNetError: 91656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.032802] LNet: Removed LNI 192.168.202.40@tcp [ 1535.045218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.088493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.108250] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1539.382807] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:35:11 (1778927711) [ 1540.262901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1541.357820] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:35:13 (1778927713) [ 1541.658571] LNet: 92695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.661524] LNetError: 92695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.673057] LNet: Removed LNI 192.168.202.40@tcp [ 1542.160576] Key type .llcrypt unregistered [ 1542.162544] Key type ._llcrypt unregistered [ 1546.545787] Key type ._llcrypt registered [ 1546.547937] Key type .llcrypt registered [ 1546.680473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1546.698638] alg: No test for adler32 (adler32-zlib) [ 1547.637679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1547.667585] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1552.560839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1556.007827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.140@tcp [ 1556.335223] LNet: 93111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.349921] LNetError: 93111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.363665] LNet: Removed LNI 192.168.202.40@tcp [ 1556.707407] Key type .llcrypt unregistered [ 1556.709939] Key type ._llcrypt unregistered [ 1564.134584] Key type ._llcrypt registered [ 1564.137089] Key type .llcrypt registered [ 1564.944806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1565.930752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1566.833072] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:35:39 (1778927739) [ 1567.802804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1568.919518] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:35:41 (1778927741) [ 1569.993992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg240-server [ 1571.213501] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:35:43 (1778927743) [ 1572.684323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg240-server [ 1573.946800] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:35:46 (1778927746) [ 1574.942368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.972815] alg: No test for adler32 (adler32-zlib) [ 1575.907927] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1576.042224] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1577.663161] Key type lgssc registered [ 1578.243648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1591.278513] Key type lgssc unregistered [ 1591.424979] LNet: 96283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.429482] LNetError: 96283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.443029] LNet: Removed LNI 192.168.202.40@tcp [ 1591.788627] Key type .llcrypt unregistered [ 1591.791491] Key type ._llcrypt unregistered [ 1599.696237] Key type ._llcrypt registered [ 1599.698063] Key type .llcrypt registered [ 1600.436363] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:36:12 (1778927772) [ 1600.650170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.657435] alg: No test for adler32 (adler32-zlib) [ 1601.487544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.548476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1601.582851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1601.620508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1601.654832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1605.337681] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:36:17 (1778927777) [ 1605.389769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.405321] LNet: 97688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.421454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.458197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.476473] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1605.971784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.982258] LNet: 97839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.984884] LNetError: 97839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.000339] LNet: Removed LNI 192.168.202.40@tcp [ 1606.010829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.046657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1606.064218] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1606.345389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.370982] LNet: Removed LNI 192.168.202.40@tcp [ 1606.390149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.431464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1606.449245] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1610.253037] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:36:22 (1778927782) [ 1610.376949] LNet: 98457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.380740] LNet: 98457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1610.390206] LNetError: 98457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.399361] LNetError: 98457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1610.422544] LNet: Removed LNI 192.168.202.40@tcp [ 1610.747641] Key type .llcrypt unregistered [ 1610.749204] Key type ._llcrypt unregistered [ 1614.489386] Key type ._llcrypt registered [ 1614.490695] Key type .llcrypt registered [ 1614.610400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1614.617245] alg: No test for adler32 (adler32-zlib) [ 1615.495512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.545082] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1615.546639] LNet: *** cfs_fail_loc=e003, val=100*** [ 1615.971572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1615.986727] Ping reply received with mapping: [ 1615.988789] found metadata reply NI[192.168.202.40@tcp] [ 1615.991540] 28 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1615.997320] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1616.000532] 5a 5a 5a 5a 5a 5a 5a 5a [ 1616.209711] LNet: 98854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.216168] LNetError: 98854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.260756] LNet: Removed LNI 192.168.202.40@tcp [ 1617.709527] Key type .llcrypt unregistered [ 1617.711859] Key type ._llcrypt unregistered [ 1624.611732] Key type ._llcrypt registered [ 1624.613286] Key type .llcrypt registered [ 1625.583631] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:36:37 (1778927797) [ 1625.906636] Key type .llcrypt unregistered [ 1625.910748] Key type ._llcrypt unregistered [ 1629.565783] Key type ._llcrypt registered [ 1629.567295] Key type .llcrypt registered [ 1629.668729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.675366] alg: No test for adler32 (adler32-zlib) [ 1630.667582] LNetError: 99899:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1630.869730] Key type .llcrypt unregistered [ 1630.872179] Key type ._llcrypt unregistered [ 1633.808644] Key type ._llcrypt registered [ 1633.810231] Key type .llcrypt registered [ 1634.630776] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:36:47 (1778927807) [ 1634.980344] Key type .llcrypt unregistered [ 1634.985080] Key type ._llcrypt unregistered [ 1638.395721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1638.400372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1640.780681] Key type ._llcrypt registered [ 1640.782351] Key type .llcrypt registered [ 1640.900291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.909912] alg: No test for adler32 (adler32-zlib) [ 1641.765210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.792750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1647.926578] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:37:00 (1778927820) [ 1648.066968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.083885] LNet: 101679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.090781] LNetError: 101679:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1648.107599] LNet: Removed LNI 10.1.2.3@tcp [ 1648.124317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.556589] Key type .llcrypt unregistered [ 1648.558581] Key type ._llcrypt unregistered [ 1652.193602] Key type ._llcrypt registered [ 1652.195677] Key type .llcrypt registered [ 1652.329970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1652.342542] alg: No test for adler32 (adler32-zlib) [ 1653.237972] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1653.305320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1653.439392] LNet: 101967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.445943] LNetError: 101967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.505648] LNet: Removed LNI 192.168.202.40@tcp [ 1655.076472] Key type .llcrypt unregistered [ 1655.080660] Key type ._llcrypt unregistered [ 1661.008976] Key type ._llcrypt registered [ 1661.011284] Key type .llcrypt registered [ 1661.856186] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:37:14 (1778927834) [ 1662.239455] Key type .llcrypt unregistered [ 1662.246475] Key type ._llcrypt unregistered [ 1666.038767] Key type ._llcrypt registered [ 1666.040607] Key type .llcrypt registered [ 1666.173068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.189752] alg: No test for adler32 (adler32-zlib) [ 1670.987768] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:37:23 (1778927843) [ 1671.069128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.092528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.137920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.153531] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1671.626073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.653100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1671.675641] LNet: Removed LNI 192.168.202.40@tcp [ 1671.869923] LNet: 103709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.153362] Key type .llcrypt unregistered [ 1672.156646] Key type ._llcrypt unregistered [ 1679.654858] Key type ._llcrypt registered [ 1679.656810] Key type .llcrypt registered [ 1680.593358] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:37:32 (1778927852) [ 1680.946614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.964758] alg: No test for adler32 (adler32-zlib) [ 1681.843341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.895984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.918944] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1682.519630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1682.539219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.40@tcp [ 1682.553947] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1682.557272] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1682.559864] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1682, 0, 0 [ 1682.562707] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1682.564925] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1682.567902] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1682.569948] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1682, 0, 0 [ 1682.573641] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1682.576116] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1682.579261] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1682.581835] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1682, 0, 1732 [ 1682.585660] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1682.587868] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1682.591091] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1682.594287] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1682, 0, 1732 [ 1682.597267] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1686.741150] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:37:39 (1778927859) [ 1687.126188] LNet: 105248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.129849] LNetError: 105248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.166232] LNet: Removed LNI 192.168.202.40@tcp [ 1688.581338] Key type .llcrypt unregistered [ 1688.584446] Key type ._llcrypt unregistered [ 1692.884385] Key type ._llcrypt registered [ 1692.886459] Key type .llcrypt registered [ 1693.037708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1693.046727] alg: No test for adler32 (adler32-zlib) [ 1693.929721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1693.946725] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1698.810330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1704.480416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.512406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1704.532609] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1705.157505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1705.309241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1705.315262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1705.318504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1707.528926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1707.546107] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1713.889524] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:38:06 (1778927886) [ 1714.026868] LNet: 106754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.035641] LNetError: 106754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.042155] LNet: Removed LNI 192.168.202.40@tcp [ 1715.044249] LNetError: 106754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.057903] LNet: Removed LNI 192.168.202.40@tcp1 [ 1715.075618] LNet: Removed LNI 10.1.2.3@tcp99 [ 1715.507551] Key type .llcrypt unregistered [ 1715.509249] Key type ._llcrypt unregistered [ 1719.588815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1719.592274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1721.918421] Key type ._llcrypt registered [ 1721.919496] Key type .llcrypt registered [ 1722.045518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1722.059067] alg: No test for adler32 (adler32-zlib) [ 1722.887727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.929208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.968382] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1723.461319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1723.479734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1724.814459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1724.853994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1724.886709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1724.905564] LNetError: 107513:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1724.942548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1724.977720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1725.206534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1725.249558] LNet: 107533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.252721] LNetError: 107533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.303265] LNet: Removed LNI 192.168.202.40@tcp [ 1726.305390] LNetError: 107533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1726.320738] LNet: Removed LNI 10.1.2.3@tcp [ 1726.717688] Key type .llcrypt unregistered [ 1726.723120] Key type ._llcrypt unregistered [ 1732.755788] Key type ._llcrypt registered [ 1732.757205] Key type .llcrypt registered [ 1733.473964] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:38:26 (1778927906) [ 1733.737987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1733.743882] alg: No test for adler32 (adler32-zlib) [ 1734.583573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.617867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.630133] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1735.059345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1735.096981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg240-client@tcp [ 1738.333605] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:38:30 (1778927910) [ 1738.405437] LNet: 109060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.410532] LNetError: 109060:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.423261] LNet: Removed LNI 192.168.202.40@tcp [ 1738.720522] Key type .llcrypt unregistered [ 1738.721674] Key type ._llcrypt unregistered [ 1741.586165] Key type ._llcrypt registered [ 1741.587280] Key type .llcrypt registered [ 1741.670432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.676702] alg: No test for adler32 (adler32-zlib) [ 1742.491926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1742.518148] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1742.539477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1745.599854] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:38:38 (1778927918) [ 1746.552380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1747.336634] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:38:40 (1778927920) [ 1748.161441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1749.202156] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:38:41 (1778927921) [ 1749.322758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.344188] LNet: 110483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.347665] LNetError: 110483:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.365645] LNet: Removed LNI 192.168.202.40@tcp [ 1749.386249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.438713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.463161] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1749.947838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1749.977186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1753.905378] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:38:46 (1778927926) [ 1753.955592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.971263] LNet: 111186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.975746] LNetError: 111186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.987699] LNet: Removed LNI 192.168.202.40@tcp [ 1754.005219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.053866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1754.073077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1754.423893] Key type .llcrypt unregistered [ 1754.427507] Key type ._llcrypt unregistered [ 1760.749448] Key type ._llcrypt registered [ 1760.751146] Key type .llcrypt registered [ 1761.542030] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:38:54 (1778927934) [ 1761.735870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.747916] alg: No test for adler32 (adler32-zlib) [ 1762.599289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.624635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.635590] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1763.070478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1763.120844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1766.502365] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:38:59 (1778927939) [ 1766.588938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.611558] LNet: 112743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.621780] LNetError: 112743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.641259] LNet: Removed LNI 192.168.202.40@tcp [ 1766.659929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.689850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1766.706444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1770.005273] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:39:02 (1778927942) [ 1770.068073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.081608] LNet: 113321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.095869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.144512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1770.591755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1770.618433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1770.660805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1770.895560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1770.921550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1770.972639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.993580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.025088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1771.199675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1771.217590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1771.250609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1771.492694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1771.511511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1775.010030] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:39:07 (1778927947) [ 1775.098685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.116978] LNet: 113958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.119369] LNet: 113958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1775.144839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.183831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.207572] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1775.631368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1775.646898] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1776.092884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1776.134254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1776.284369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.293807] LNetError: 114239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.297146] LNet: Removed LNI 192.168.202.40@tcp [ 1776.314407] LNet: Removed LNI 192.168.202.40@tcp2 [ 1776.328715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.368401] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1776.373300] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1776.377224] LNet: 114245:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.748819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1776.868604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.887766] LNet: Removed LNI 192.168.202.40@tcp [ 1777.889460] LNetError: 114282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.893132] LNetError: 114282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1777.908431] LNet: Removed LNI 192.168.202.40@tcp2 [ 1777.933983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.980361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.993889] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1778.453877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1778.470360] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1779.001502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1779.022980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1779.068860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1779.227062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.244399] LNet: 114567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.252461] LNet: 114567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1780.319227] LNet: Removed LNI 192.168.202.40@tcp [ 1780.321688] LNetError: 114567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.330882] LNetError: 114567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1780.348031] LNet: Removed LNI 192.168.202.40@tcp2 [ 1780.379602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.440446] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1780.445935] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1780.450559] LNet: 114573:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.789663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1780.934766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.950183] LNet: Removed LNI 192.168.202.40@tcp [ 1781.990872] LNet: Removed LNI 192.168.202.40@tcp2 [ 1782.011974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.049136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.063110] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1782.576141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1782.596580] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1783.186214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1783.225213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1783.367602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.380745] LNet: 114889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.384486] LNet: 114889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1784.415321] LNet: Removed LNI 192.168.202.40@tcp [ 1784.420702] LNetError: 114889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.422756] LNetError: 114889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1784.433835] LNet: Removed LNI 192.168.202.40@tcp2 [ 1784.454675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.485769] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1784.488881] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1784.491864] LNet: 114896:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.831873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1784.974597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.016258] LNet: Removed LNI 192.168.202.40@tcp [ 1786.031453] LNet: Removed LNI 192.168.202.40@tcp2 [ 1786.065191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.106503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.124987] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1786.618850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1786.636188] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1787.133141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1787.154593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1787.180449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1787.292079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.319136] LNet: Removed LNI 192.168.202.40@tcp [ 1789.352836] LNet: Removed LNI 192.168.202.40@tcp2 [ 1789.382914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.437537] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1789.444582] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1789.448954] LNet: 115224:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.738123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1789.897935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.913130] LNet: Removed LNI 192.168.202.40@tcp [ 1790.953630] LNet: Removed LNI 192.168.202.40@tcp2 [ 1790.974775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.027017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.043547] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1791.530477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1791.548343] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1792.129180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1792.153532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1792.193598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1792.520445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.532520] LNet: 115557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.536275] LNet: 115557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1792.543043] LNetError: 115557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.546734] LNetError: 115557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1792.548935] LNet: Removed LNI 192.168.202.40@tcp [ 1792.557496] LNet: Removed LNI 192.168.202.40@tcp2 [ 1792.569740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.607734] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1792.612976] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1792.858446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1793.079187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.143256] LNet: Removed LNI 192.168.202.40@tcp [ 1794.157942] LNet: Removed LNI 192.168.202.40@tcp2 [ 1794.180965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.233624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.252508] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1794.697237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1794.706920] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1795.251417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1795.280912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1795.303584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1795.343977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1795.574840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.639141] LNet: Removed LNI 192.168.202.40@tcp [ 1797.670726] LNet: Removed LNI 192.168.202.40@tcp2 [ 1797.690190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.732837] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1797.737283] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1797.741324] LNet: 115923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1797.748883] LNet: 115923:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1798.030216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1802.031426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1803.090419] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:39:35 (1778927975) [ 1803.258960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.282287] LNet: Removed LNI 192.168.202.40@tcp [ 1804.334315] LNet: Removed LNI 192.168.202.40@tcp2 [ 1804.346409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.573804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.594922] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1805.025896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1805.047662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1805.719692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.732724] LNet: Removed LNI 192.168.202.40@tcp [ 1805.738753] LNet: Removed LNI 10.1.2.3@tcp [ 1805.750522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.804607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1805.827920] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1805.833411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1806.062792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1806.077851] LNet: Removed LNI 192.168.202.40@tcp [ 1806.087300] LNet: Removed LNI 10.1.2.3@tcp [ 1809.701918] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:39:42 (1778927982) [ 1809.854740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.865708] LNet: 117554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.868740] LNet: 117554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1809.881672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.963240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1810.033096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1810.035530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1810.037994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1812.228138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.243314] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1812.671837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1812.682899] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1814.034554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1814.085952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1814.152584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.40@tcp -m PUT -e local_timeout -r 1 [ 1814.198963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1814.253273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1814.257102] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1814.480544] LNetError: 117620:0:(lib-move.c:4710:lnet_parse()) 192.168.202.40@tcp, src 192.168.202.40@tcp: Dropping PUT (error -108 looking up sender) [ 1815.526555] LNetError: 118077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.534894] LNetError: 118077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1815.542827] LNet: Removed LNI 192.168.202.40@tcp [ 1815.549958] LNet: Removed LNI 10.1.2.3@tcp [ 1815.891450] Key type .llcrypt unregistered [ 1815.892981] Key type ._llcrypt unregistered [ 1823.679340] Key type ._llcrypt registered [ 1823.680544] Key type .llcrypt registered [ 1824.584701] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:39:56 (1778927996) [ 1824.861692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.869796] alg: No test for adler32 (adler32-zlib) [ 1825.742621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.020731] LNet: 118930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.479402] Key type .llcrypt unregistered [ 1826.482306] Key type ._llcrypt unregistered [ 1829.879348] Key type ._llcrypt registered [ 1829.882327] Key type .llcrypt registered [ 1830.003977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1830.012209] alg: No test for adler32 (adler32-zlib) [ 1830.878806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1830.902910] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1835.428502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1835.539938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1835.866138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 1835.997616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1836.019889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1836.044255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1836.068147] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1836.071371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1836.110184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1836.137257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1836.161681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1836.183996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1836.193958] LNet: 119395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1836.205651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.228220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.251516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.274447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.293595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.320377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1836.716922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1836.878392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1836.912692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1836.947292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1836.969602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1836.997311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1837.022305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1837.044885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1837.058418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1837.069138] LNet: 119513:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1837.089234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1837.110751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1837.137374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1837.174676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1837.219379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1837.254450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.858477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 1838.031311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.060948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1838.081371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.103213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1838.123689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1838.150101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1838.173872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1838.197116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1838.207946] LNet: 119630:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.221469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1838.245268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1838.268577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.289217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.314663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.340266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1838.826212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1838.986859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.013941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1839.037766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.076496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.111733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.135612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.169991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.230129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.272363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.321405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.352394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.384133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1839.408426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1839.426210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1839.715784] LNet: 119793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.722828] LNetError: 119793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.735751] LNet: Removed LNI 192.168.202.40@tcp [ 1840.082749] Key type .llcrypt unregistered [ 1840.084087] Key type ._llcrypt unregistered [ 1851.147413] Key type ._llcrypt registered [ 1851.149246] Key type .llcrypt registered [ 1851.979959] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:40:24 (1778928024) [ 1852.254483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.265267] alg: No test for adler32 (adler32-zlib) [ 1853.127151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.382939] LNet: 120680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.795594] Key type .llcrypt unregistered [ 1853.800641] Key type ._llcrypt unregistered [ 1858.320779] Key type ._llcrypt registered [ 1858.323510] Key type .llcrypt registered [ 1858.486452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1858.500512] alg: No test for adler32 (adler32-zlib) [ 1859.368092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1859.386049] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1864.749666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1870.162840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.201103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1870.220330] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1870.837189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1871.213468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 1871.388647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1871.411587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1871.428433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1871.442848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1871.461380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1871.488612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1871.518785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1871.554889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1871.578026] LNet: 121331:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1871.601652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1871.623510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1871.651471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1871.672985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1871.700342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1871.711267] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1871.732121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1872.148686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1872.323214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1872.355445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1872.381205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1872.401618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1872.423706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1872.447529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1872.476893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1872.506791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1872.519207] LNet: 121448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1872.533813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1872.560916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.584598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.615259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.640097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.669750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1873.171939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 1873.275628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1873.289172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1873.303695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1873.319671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1873.336813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1873.354895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1873.374846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1873.396370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1873.422215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1873.447696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1873.472674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1873.507622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1873.527496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1873.543875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1873.979649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1874.134779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1874.187048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1874.213558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1874.237749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1874.261720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1874.283382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1874.313270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1874.341990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1874.355691] LNet: 121682:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1874.361558] LNet: 121682:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1874.374196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1874.396223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1874.421287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1874.440486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1874.463623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1874.490576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1877.854918] LNet: 121763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.863874] LNetError: 121763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.874634] LNet: Removed LNI 192.168.202.40@tcp [ 1877.888510] LNet: Removed LNI 192.168.202.40@tcp1 [ 1878.213116] Key type .llcrypt unregistered [ 1878.214806] Key type ._llcrypt unregistered [ 1888.360936] Key type ._llcrypt registered [ 1888.362312] Key type .llcrypt registered [ 1889.066744] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:41:01 (1778928061) [ 1893.221979] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:41:05 (1778928065) [ 1897.681856] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:41:10 (1778928070) [ 1901.865537] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:41:14 (1778928074) [ 1906.165776] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:41:18 (1778928078) [ 1910.433409] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:41:22 (1778928082) [ 1915.026965] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:41:27 (1778928087) [ 1919.464266] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:41:32 (1778928092) [ 1923.550780] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:41:36 (1778928096) [ 1927.517338] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:41:40 (1778928100) [ 1932.040465] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:41:44 (1778928104) [ 1936.823015] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:41:49 (1778928109) [ 1941.270139] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:41:53 (1778928113) [ 1946.000596] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:41:58 (1778928118) [ 1950.557218] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:42:03 (1778928123) [ 1954.559574] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:42:07 (1778928127) [ 1958.567836] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:42:11 (1778928131) [ 1963.153815] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:42:15 (1778928135) [ 1963.366085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.376612] alg: No test for adler32 (adler32-zlib) [ 1964.228453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.307873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.325768] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1965.004310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.015507] LNet: 132464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.020265] LNetError: 132464:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.039295] LNet: Removed LNI 192.168.202.40@tcp [ 1965.053120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.088502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1965.104167] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1968.885913] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:42:21 (1778928141) [ 1972.979709] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:42:25 (1778928145) [ 1977.331388] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:42:29 (1778928149) [ 1981.487284] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:42:34 (1778928154) [ 1985.321092] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:42:37 (1778928157) [ 1989.387689] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:42:41 (1778928161) [ 1993.791721] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:42:46 (1778928166) [ 1998.197870] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:42:50 (1778928170) [ 2002.300736] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:42:54 (1778928174) [ 2006.653671] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:42:59 (1778928179) [ 2010.654178] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:43:03 (1778928183) [ 2014.955514] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:43:07 (1778928187) [ 2018.804465] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1718 sec ========== 06:43:11 (1778928191) [ 2019.017508] LNet: 139924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.021251] LNetError: 139924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.030152] LNet: Removed LNI 192.168.202.40@tcp [ 2019.448448] Key type .llcrypt unregistered [ 2019.449819] Key type ._llcrypt unregistered [ 2026.413215] Key type ._llcrypt registered [ 2026.423149] Key type .llcrypt registered [ 2026.724825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.736110] alg: No test for adler32 (adler32-zlib) [ 2027.734515] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2027.891022] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2029.519178] Key type lgssc registered [ 2030.271147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.096538] Lustre: Mounted lustre-client [ 2071.591161] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2075.652133] LustreError: 142367:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9523c3095800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2075.678378] Lustre: Unmounted lustre-client [ 2087.186623] Key type lgssc unregistered [ 2087.347322] LNet: 142847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.351403] LNetError: 142847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.365046] LNet: Removed LNI 192.168.202.40@tcp [ 2087.673139] Key type .llcrypt unregistered [ 2087.674327] Key type ._llcrypt unregistered