[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450038812 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003158] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.007895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010180] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013051] SELinux: Initializing. [ 0.014085] *** VALIDATE selinux *** [ 0.023022] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028140] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031129] *** VALIDATE tmpfs *** [ 0.033135] *** VALIDATE proc *** [ 0.034180] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.037059] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044196] debug: unmapping init [mem 0xffffffffa9259000-0xffffffffa9260fff] [ 0.047181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054008] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056316] rcu: Hierarchical SRCU implementation. [ 0.058589] smp: Bringing up secondary CPUs ... [ 0.059615] x86: Booting SMP configuration: [ 0.060039] .... node #0, CPUs: #1 #2 #3 [ 0.063629] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138015] node 0 deferred pages initialised in 71ms [ 0.141120] devtmpfs: initialized [ 0.142293] x86/mm: Memory block size: 128MB [ 0.145846] gcov: version magic: 0x41383552 [ 0.148231] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154360] pinctrl core: initialized pinctrl subsystem [ 0.156250] [ 0.156815] ************************************************************* [ 0.159017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161016] ** ** [ 0.163014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.168016] ** This means that this kernel is built to expose internal ** [ 0.170012] ** IOMMU data structures, which may compromise security on ** [ 0.172031] ** your system. ** [ 0.174018] ** ** [ 0.176014] ** If you see this message and you are not debugging the ** [ 0.178014] ** kernel, report this immediately to your vendor! ** [ 0.181016] ** ** [ 0.183013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185019] ************************************************************* [ 0.189207] NET: Registered protocol family 16 [ 0.191503] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200107] cpuidle: using governor menu [ 0.201608] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204525] PCI: Using configuration type 1 for base access [ 0.206132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214178] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216076] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218198] cryptd: max_cpu_qlen set to 1000 [ 0.221223] ACPI: Added _OSI(Module Device) [ 0.222011] ACPI: Added _OSI(Processor Device) [ 0.223015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230659] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236049] ACPI: Interpreter enabled [ 0.236948] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238013] ACPI: Using IOAPIC for interrupt routing [ 0.239095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241527] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250625] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253137] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.262557] acpiphp: Slot [2] registered [ 0.264124] acpiphp: Slot [5] registered [ 0.266100] acpiphp: Slot [6] registered [ 0.267044] acpiphp: Slot [3] registered [ 0.267912] acpiphp: Slot [4] registered [ 0.268066] acpiphp: Slot [7] registered [ 0.268974] acpiphp: Slot [8] registered [ 0.270056] acpiphp: Slot [9] registered [ 0.270944] acpiphp: Slot [10] registered [ 0.272067] acpiphp: Slot [11] registered [ 0.273014] acpiphp: Slot [12] registered [ 0.273918] acpiphp: Slot [13] registered [ 0.274079] acpiphp: Slot [14] registered [ 0.275051] acpiphp: Slot [15] registered [ 0.275976] acpiphp: Slot [16] registered [ 0.277099] acpiphp: Slot [17] registered [ 0.278052] acpiphp: Slot [18] registered [ 0.279075] acpiphp: Slot [19] registered [ 0.279971] acpiphp: Slot [20] registered [ 0.281104] acpiphp: Slot [21] registered [ 0.282075] acpiphp: Slot [22] registered [ 0.282840] acpiphp: Slot [23] registered [ 0.284058] acpiphp: Slot [24] registered [ 0.284964] acpiphp: Slot [25] registered [ 0.285068] acpiphp: Slot [26] registered [ 0.287088] acpiphp: Slot [27] registered [ 0.288097] acpiphp: Slot [28] registered [ 0.289010] acpiphp: Slot [29] registered [ 0.289926] acpiphp: Slot [30] registered [ 0.291075] acpiphp: Slot [31] registered [ 0.291966] PCI host bridge to bus 0000:00 [ 0.292016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.302168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303910] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305971] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312802] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.315538] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323570] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325606] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329711] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.342612] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.347011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.353287] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.358018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.364022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383826] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.408019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417971] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.420431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.423422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.425219] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.426130] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.430029] iommu: Default domain type: Passthrough [ 0.431473] SCSI subsystem initialized [ 0.433122] ACPI: bus type USB registered [ 0.435170] usbcore: registered new interface driver usbfs [ 0.436099] usbcore: registered new interface driver hub [ 0.439113] usbcore: registered new device driver usb [ 0.440213] pps_core: LinuxPPS API ver. 1 registered [ 0.442008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.444062] PTP clock support registered [ 0.446089] EDAC MC: Ver: 3.0.0 [ 0.448092] PCI: Using ACPI for IRQ routing [ 0.449565] NetLabel: Initializing [ 0.450010] NetLabel: domain hash size = 128 [ 0.451011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.452080] NetLabel: unlabeled traffic allowed by default [ 0.455039] vgaarb: loaded [ 0.457272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.458007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.462297] clocksource: Switched to clocksource kvm-clock [ 0.567831] VFS: Disk quotas dquot_6.6.0 [ 0.569536] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.572240] *** VALIDATE ramfs *** [ 0.573541] *** VALIDATE hugetlbfs *** [ 0.575125] pnp: PnP ACPI init [ 0.577135] pnp: PnP ACPI: found 6 devices [ 0.592168] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.595711] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.598121] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.600463] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.603031] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.605501] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.608176] NET: Registered protocol family 2 [ 0.610801] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.615775] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.619648] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.624824] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.627962] TCP: Hash tables configured (established 65536 bind 65536) [ 0.630446] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.633807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.636595] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.639495] NET: Registered protocol family 1 [ 0.642183] RPC: Registered named UNIX socket transport module. [ 0.643439] RPC: Registered udp transport module. [ 0.644684] RPC: Registered tcp transport module. [ 0.646429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.648082] NET: Registered protocol family 44 [ 0.649157] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.650507] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.651728] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.653334] PCI: CLS 0 bytes, default 64 [ 0.655334] Unpacking initramfs... [ 2.018993] debug: unmapping init [mem 0xffff8ba43cc64000-0xffff8ba43ffcffff] [ 2.022684] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.025591] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.029093] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.519705] Initialise system trusted keyrings [ 2.521653] Key type blacklist registered [ 2.523773] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.532165] zbud: loaded [ 2.535885] *** VALIDATE nfs *** [ 2.537237] *** VALIDATE nfs4 *** [ 2.539399] pstore: using deflate compression [ 2.542886] Platform Keyring initialized [ 2.636783] NET: Registered protocol family 38 [ 2.638657] Key type asymmetric registered [ 2.640155] Asymmetric key parser 'x509' registered [ 2.641920] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.644949] io scheduler mq-deadline registered [ 2.646630] io scheduler kyber registered [ 2.648313] io scheduler bfq registered [ 2.650075] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.653149] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.656151] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.658894] ACPI: Power Button [PWRF] [ 2.664517] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.669627] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.677885] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.704230] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.731700] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.735859] Non-volatile memory driver v1.3 [ 2.737358] Linux agpgart interface v0.103 [ 2.769502] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.772661] vda: detected capacity change from 0 to 76677120 [ 2.785894] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.788431] vdb: detected capacity change from 0 to 1073741824 [ 2.794665] libphy: Fixed MDIO Bus: probed [ 2.804269] usbcore: registered new interface driver usbserial_generic [ 2.806476] usbserial: USB Serial support registered for generic [ 2.808949] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.813512] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.815051] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.817351] mousedev: PS/2 mouse device common for all mice [ 2.820326] rtc_cmos 00:05: RTC can wake from S4 [ 2.822600] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.824084] rtc_cmos 00:05: registered as rtc0 [ 2.827608] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.828478] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.830217] intel_pstate: CPU model not supported [ 2.834318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.838536] hid: raw HID events driver (C) Jiri Kosina [ 2.840225] usbcore: registered new interface driver usbhid [ 2.841799] usbhid: USB HID core driver [ 2.843201] drop_monitor: Initializing network drop monitor service [ 2.845095] Initializing XFRM netlink socket [ 2.846953] NET: Registered protocol family 10 [ 2.849564] Segment Routing with IPv6 [ 2.850743] NET: Registered protocol family 17 [ 2.852395] mpls_gso: MPLS GSO support [ 2.856909] RAS: Correctable Errors collector initialized. [ 2.858516] AVX version of gcm_enc/dec engaged. [ 2.859943] AES CTR mode by8 optimization enabled [ 2.936152] sched_clock: Marking stable (2936126027, 0)->(3770146225, -834020198) [ 2.939137] registered taskstats version 1 [ 2.941393] Loading compiled-in X.509 certificates [ 2.942765] zswap: loaded using pool lzo/zbud [ 2.963423] Key type big_key registered [ 2.974689] Key type encrypted registered [ 2.975804] ima: No TPM chip found, activating TPM-bypass! [ 2.978296] ima: Allocated hash algorithm: sha1 [ 2.979489] ima: No architecture policies found [ 2.981097] evm: Initialising EVM extended attributes: [ 2.982520] evm: security.selinux [ 2.983488] evm: security.ima [ 2.984225] evm: security.capability [ 2.985154] evm: HMAC attrs: 0x1 [ 2.987867] rtc_cmos 00:05: setting system clock to 2026-09-05 14:46:39 UTC (1788619599) [ 2.994066] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 2.997330] debug: unmapping init [mem 0xffffffffa8f82000-0xffffffffa9258fff] [ 3.006088] Write protecting the kernel read-only data: 28672k [ 3.009875] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 3.012575] debug: unmapping init [mem 0xffffffffa7f14000-0xffffffffa7ffffff] [ 3.041204] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.046953] systemd[1]: Detected virtualization kvm. [ 3.048192] systemd[1]: Detected architecture x86-64. [ 3.049265] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.073380] systemd[1]: No hostname configured. [ 3.075173] systemd[1]: Set hostname to . [ 3.076800] random: systemd: uninitialized urandom read (16 bytes read) [ 3.078607] systemd[1]: Initializing machine ID from random generator. [ 3.120333] random: ln: uninitialized urandom read (6 bytes read) [ 3.191986] random: systemd: uninitialized urandom read (16 bytes read) [ 3.195102] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.201963] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.209374] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.884796] device-mapper: uevent: version 1.0.3 [ 3.887339] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [[ 4.671758] random: fast init done  OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 4.718606] virtio_net virtio0 ens2: renamed from eth0 [ 4.844833] scsi host0: ata_piix [ 4.861341] scsi host1: ata_piix [ 4.863055] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.865479] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.187184] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.492783] random: crng init done [ 9.494326] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.915150] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.108086] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.365679] SELinux: Disabled at runtime. [ 11.427513] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.439616] systemd[1]: Detected virtualization kvm. [ 11.442474] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.958719] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.961994] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.967166] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.971317] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.975236] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.983928] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.993655] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [[ 12.088351] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.435641] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.717218] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.786494] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.885379] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.901832] EDAC sbridge: Ver: 1.1.2 [ 14.659412] Key type dns_resolver registered [ 15.462899] NFS: Registering the id_resolver key type [ 15.471306] Key type id_resolver registered [ 15.474340] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg201-client login: [ 35.138387] hrtimer: interrupt took 1505531 ns [ 91.952372] libcfs: loading out-of-tree module taints kernel. [ 92.469917] Key type ._llcrypt registered [ 92.488588] Key type .llcrypt registered [ 93.019537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 93.042711] alg: No test for adler32 (adler32-zlib) [ 94.460646] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 95.388092] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 97.175196] Key type lgssc registered [ 99.033232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 244.526407] Lustre: Mounted lustre-client - version 2.17.58_3_g10061d6 [ 249.286918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.661378] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing check_logdir /tmp/testlogs/ [ 265.291746] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing yml_node [ 269.575898] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 270.313831] Lustre: lustre-OST0000-osc-ffff8ba48c1cf000: disconnect after 23s idle [ 272.223452] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 274.540482] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 276.373149] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 10:51:11 EDT 2026 [ 292.840786] Lustre: Unmounted lustre-client [ 312.901771] Key type lgssc unregistered [ 313.185672] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.196701] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.218340] LNet: Removed LNI 192.168.202.1@tcp [ 314.159926] Key type .llcrypt unregistered [ 314.161934] Key type ._llcrypt unregistered [ 341.021686] Key type ._llcrypt registered [ 341.024312] Key type .llcrypt registered [ 341.335567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.371293] alg: No test for adler32 (adler32-zlib) [ 342.317513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 342.365921] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 342.380279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 346.693224] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 346.922224] LNet: 9289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.942266] LNetError: 9289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.961871] LNet: Removed LNI 192.168.202.1@tcp [ 347.702873] Key type .llcrypt unregistered [ 347.709669] Key type ._llcrypt unregistered [ 355.781219] Key type ._llcrypt registered [ 355.785426] Key type .llcrypt registered [ 357.799629] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:52:32 (1788619952) [ 358.177855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 358.190361] alg: No test for adler32 (adler32-zlib) [ 359.179446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.303564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.341222] LNet: 9715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.379924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.519666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.536238] LNet: 9722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 367.719911] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:52:42 (1788619962) [ 367.897089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.912125] LNet: 10295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.943463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.042771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 375.523475] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:52:50 (1788619970) [ 375.661862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.688929] LNet: 10881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.720494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.835570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 383.497266] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:52:58 (1788619978) [ 384.039778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.063563] LNet: 11471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.089554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.208938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 392.809610] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:53:07 (1788619987) [ 392.935294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.970439] LNet: 12049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.017108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.137909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 394.012463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 402.906952] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:53:17 (1788619997) [ 403.158127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.187852] LNet: 12647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.232677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.374582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 404.087863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 404.880404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 412.014219] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:53:27 (1788620007) [ 412.178609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.233705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.334512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 420.468865] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:53:35 (1788620015) [ 420.604942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.635949] LNet: 13822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.649720] LNet: 13822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.718940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.862597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 420.938199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.625792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 421.707388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 422.235690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 422.305305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 423.367984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 423.444736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 423.998368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 424.029489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 424.374704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 424.416652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.747339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 424.777991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.132538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 425.154744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 425.448879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 425.492259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.180338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 426.236901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 427.079654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 427.153499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.208809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 428.278525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 428.915862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 428.968118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 436.291792] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:53:51 (1788620031) [ 436.532884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.585563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.712165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 436.776586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 443.951599] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:53:59 (1788620039) [ 444.179565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.219422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.302490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 444.343191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 452.376623] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:54:07 (1788620047) [ 452.534108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.589119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.748961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 452.792983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 460.529489] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:54:15 (1788620055) [ 460.776796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.800272] LNet: 16271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.809435] LNet: 16271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 460.855302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.987562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 461.065078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 468.805968] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:54:23 (1788620063) [ 468.920843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.996212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.101860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 469.141077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 475.901827] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:54:31 (1788620071) [ 476.176689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.261615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.362378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 476.417687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 483.315548] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:54:38 (1788620078) [ 483.609078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.665549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.758885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.194235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.149878] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:54:46 (1788620086) [ 492.308717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.362831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.596465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.753306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.670388] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:54:55 (1788620095) [ 500.992799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.075828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.311238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.772917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.398208] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:55:04 (1788620104) [ 509.525779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.601716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.243974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.596514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.956531] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:55:12 (1788620112) [ 518.076065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.148423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.281498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.321742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.571343] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:55:21 (1788620121) [ 527.645528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.672608] LNet: 21100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.677343] LNet: 21100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 527.711756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.804631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.849504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 535.931360] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:55:30 (1788620130) [ 536.058358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.137829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.246244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 544.058724] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:55:39 (1788620139) [ 544.251419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.341596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.473897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 553.638371] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:55:48 (1788620148) [ 553.990163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.040319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.216617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 554.833725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 561.761610] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:55:56 (1788620156) [ 562.046958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.129389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.230991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 562.266411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 569.269095] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:56:04 (1788620164) [ 569.454975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.519205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.580942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 569.619160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 575.810283] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:56:11 (1788620171) [ 575.949700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.997383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.059567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 576.085535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 583.069223] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:56:18 (1788620178) [ 583.191660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.230278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.312675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.338406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 589.781443] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:56:25 (1788620185) [ 589.887774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.947153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.037643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 590.101905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 590.157796] LNetError: 25832:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 590.227508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 590.256298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 595.640456] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:56:30 (1788620190) [ 595.761333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.802280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.869599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WMnIQexV7O [ 595.945623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WMnIQexV7O [ 596.003750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.WMnIQexV7O [ 602.103079] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:56:37 (1788620197) [ 602.235038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.303981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.375829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 602.408793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 602.478216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 608.018297] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:56:43 (1788620203) [ 608.138376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.189331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.329792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.371107] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 609.858212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 611.374208] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 613.327163] Key type lgssc registered [ 614.657333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 617.924272] LNetError: 27590:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.1@tcp99 has no available nets [ 620.944565] Key type lgssc unregistered [ 621.394431] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.437915] LNet: Removed LNI 192.168.202.1@tcp [ 623.224099] Key type .llcrypt unregistered [ 623.231816] Key type ._llcrypt unregistered [ 630.429917] Key type ._llcrypt registered [ 630.431893] Key type .llcrypt registered [ 631.543385] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:57:06 (1788620226) [ 632.076538] Key type .llcrypt unregistered [ 632.083249] Key type ._llcrypt unregistered [ 639.318675] Key type ._llcrypt registered [ 639.321497] Key type .llcrypt registered [ 639.613993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.634285] alg: No test for adler32 (adler32-zlib) [ 640.532977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.592408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 640.609942] LNet: 29608:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 640.616726] LNet: 29608:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 640.622655] LNet: 29608:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 640.634633] LNet: Message forwarding will be enabled in 2s [ 643.743387] LNet: Message forwarding enabled [ 644.227469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.270764] LNet: Message forwarding will stop in 2s to 6s [ 647.842842] LNet: Message forwarding disabled [ 648.799449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.827100] LNet: 29706:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 648.834258] LNet: 29706:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 648.842432] LNet: 29706:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 648.871164] LNet: Message forwarding will be enabled in 2s [ 649.081429] LNet: 29719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.779639] Key type .llcrypt unregistered [ 649.784792] Key type ._llcrypt unregistered [ 661.555504] Key type ._llcrypt registered [ 661.557399] Key type .llcrypt registered [ 662.609514] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:57:37 (1788620257) [ 662.963715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.976916] alg: No test for adler32 (adler32-zlib) [ 663.872220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.939363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 669.332696] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:57:44 (1788620264) [ 669.717832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.757905] LNet: 31119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.782371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.864641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 669.890505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 669.944190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 669.999772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 670.052362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 670.106925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 670.140903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.201084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 670.235540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 670.278439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 670.330386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 670.379412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 670.439887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 670.486601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 670.534356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 670.603320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 670.658291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 670.687167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.715670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 670.734484] LNetError: 31162:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 677.147781] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:57:52 (1788620272) [ 677.317667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.359635] LNet: 31732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.409731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.495201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.239843] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:57:58 (1788620278) [ 684.725843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg201-server [ 686.315660] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:58:01 (1788620281) [ 686.689766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.719708] LNet: 32665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.752074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.848633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.865776] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 687.685522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[2-6]@tcp [ 687.710504] LNet: 32798:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 687.774349] LNetError: 32669:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.3@tcp is being used as a gateway but routing feature is not turned on [ 687.797655] LNetError: 32669:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.3@tcp has gone from up to down [ 694.549703] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:58:09 (1788620289) [ 696.548265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg201-server [ 698.236083] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:58:13 (1788620293) [ 698.410759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.440501] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.445405] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.485827] LNet: Removed LNI 192.168.202.1@tcp [ 698.511257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.567938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.584329] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 699.109195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[2-7/2]@tcp [ 699.129749] LNet: 33851:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 699.136556] LNet: 33851:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 699.154517] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.4@tcp is being used as a gateway but routing feature is not turned on [ 699.173525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[2-7/2]@tcp [ 699.175436] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 699.175461] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.4@tcp has gone from up to down [ 699.175465] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 705.845186] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:58:21 (1788620301) [ 705.946252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.994415] LNet: 34437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.005283] LNetError: 34437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.042729] LNet: Removed LNI 192.168.202.1@tcp [ 706.131710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.234214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 706.294665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.333376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.412971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.472727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.515487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.605259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 706.713743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.741932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.789458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.860277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.896212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.940971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 707.013280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.035477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.104406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.193092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.225763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.270605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 707.323660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.351982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.406388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.491421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.522412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.574988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 707.655963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.703280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.817372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.879611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.920317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.013597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 708.091897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.143463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.217779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.939959] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:58:30 (1788620310) [ 716.800056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg201-server [ 718.697300] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:58:33 (1788620313) [ 720.633529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg201-server [ 722.113764] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:58:37 (1788620317) [ 722.494856] LNet: 35844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.506970] LNet: 35844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 722.973413] Key type .llcrypt unregistered [ 722.975807] Key type ._llcrypt unregistered [ 729.292131] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 729.552214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 729.561736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 729.575509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 732.329690] Key type ._llcrypt registered [ 732.337505] Key type .llcrypt registered [ 732.642380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.697759] alg: No test for adler32 (adler32-zlib) [ 733.600653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.696131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.723035] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 734.440880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.462501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 736.737640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 736.765587] LNet: Removed LNI 10.1.2.3@tcp [ 736.881338] LNet: 36561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.885810] LNetError: 36561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.905843] LNet: Removed LNI 192.168.202.1@tcp [ 737.597437] Key type .llcrypt unregistered [ 737.599429] Key type ._llcrypt unregistered [ 750.839774] Key type ._llcrypt registered [ 750.841392] Key type .llcrypt registered [ 752.277369] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:59:07 (1788620347) [ 752.784218] Key type .llcrypt unregistered [ 752.786883] Key type ._llcrypt unregistered [ 759.421434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.530798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.546221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.558344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.362551] Key type ._llcrypt registered [ 762.369774] Key type .llcrypt registered [ 762.579129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.600853] alg: No test for adler32 (adler32-zlib) [ 763.621701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.723269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.745108] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 764.761442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.781296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.807568] LNet: 38048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.815904] LNetError: 38048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 767.821149] LNet: Removed LNI 10.1.2.3@tcp [ 767.828807] LNet: Removed LNI 192.168.202.1@tcp [ 768.515441] Key type .llcrypt unregistered [ 768.517085] Key type ._llcrypt unregistered [ 782.033494] Key type ._llcrypt registered [ 782.037184] Key type .llcrypt registered [ 783.549156] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:59:38 (1788620378) [ 784.192578] Key type .llcrypt unregistered [ 784.194704] Key type ._llcrypt unregistered [ 791.116626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.237741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.242117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.248206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.948174] Key type ._llcrypt registered [ 793.956420] Key type .llcrypt registered [ 794.253854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.268400] alg: No test for adler32 (adler32-zlib) [ 795.138401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.201390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.237457] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 797.468409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.487422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.833431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 799.850641] LNet: Removed LNI 10.1.2.3@tcp [ 799.869941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 799.918909] LNet: Removed LNI 10.1.2.31@tcp [ 799.960984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 799.993646] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.790836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 800.810053] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 801.842311] LNet: 39957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.852131] LNetError: 39957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 801.860523] LNet: Removed LNI 10.1.2.31@tcp [ 801.879176] LNet: Removed LNI 10.1.2.3@tcp [ 802.571155] Key type .llcrypt unregistered [ 802.577218] Key type ._llcrypt unregistered [ 816.657571] Key type ._llcrypt registered [ 816.659950] Key type .llcrypt registered [ 818.316859] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:00:13 (1788620413) [ 818.810299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 818.823438] alg: No test for adler32 (adler32-zlib) [ 819.739542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.804722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.837917] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 820.748158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.763790] LNet: 40912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.767698] LNetError: 40912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.778217] LNet: Removed LNI 192.168.202.1@tcp [ 820.794709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.903678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.1@tcp [ 820.961419] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 821.777392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.1@tcp [ 821.822495] LNet: Removed LNI 192.168.202.1@tcp [ 821.911617] LNet: 41047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.674747] Key type .llcrypt unregistered [ 822.679480] Key type ._llcrypt unregistered [ 835.486125] Key type ._llcrypt registered [ 835.489571] Key type .llcrypt registered [ 836.931817] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:00:31 (1788620431) [ 838.662752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg201-server [ 840.583389] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:00:35 (1788620435) [ 841.072431] Key type .llcrypt unregistered [ 841.074524] Key type ._llcrypt unregistered [ 848.132522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.420991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.431530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.446603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.173269] Key type ._llcrypt registered [ 851.175170] Key type .llcrypt registered [ 851.445991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.473348] alg: No test for adler32 (adler32-zlib) [ 852.552248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.650972] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 853.492542] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 855.473530] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.487649] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.493864] LNet: Removed LNI 192.168.202.1@tcp1 [ 855.517379] LNet: Removed LNI 10.1.2.3@tcp2 [ 856.213485] Key type .llcrypt unregistered [ 856.218631] Key type ._llcrypt unregistered [ 868.635329] Key type ._llcrypt registered [ 868.641967] Key type .llcrypt registered [ 869.940526] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:01:05 (1788620465) [ 870.457360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 870.478125] alg: No test for adler32 (adler32-zlib) [ 871.382884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.440218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.463581] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 872.220505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.239754] LNet: 43861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.246913] LNetError: 43861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.262948] LNet: Removed LNI 192.168.202.1@tcp [ 872.284728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.360418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 872.386135] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 878.602524] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:01:14 (1788620474) [ 878.768521] LNet: 44449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.792170] LNetError: 44449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.822500] LNet: Removed LNI 192.168.202.1@tcp [ 879.314395] Key type .llcrypt unregistered [ 879.319182] Key type ._llcrypt unregistered [ 885.437249] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.641038] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.646125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.650434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.305452] Key type ._llcrypt registered [ 888.307803] Key type .llcrypt registered [ 888.437262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.454975] alg: No test for adler32 (adler32-zlib) [ 889.383989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.421930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.459770] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 890.028970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 890.048701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 891.781379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.800574] LNet: 45185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.808688] LNetError: 45185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.814851] LNet: Removed LNI 10.1.2.3@tcp [ 891.830211] LNet: Removed LNI 192.168.202.1@tcp [ 891.846497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.915212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 891.946621] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 891.952370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.344056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.369757] LNet: 45212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.377454] LNetError: 45212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.382192] LNetError: 45212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 892.386700] LNet: Removed LNI 10.1.2.3@tcp [ 892.395723] LNet: Removed LNI 192.168.202.1@tcp [ 892.432576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.503811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.523279] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 892.527807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.934727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 892.949544] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 893.934875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.979681] LNet: 45374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.991479] LNetError: 45374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.002175] LNetError: 45374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 894.009804] LNet: Removed LNI 10.1.2.3@tcp [ 894.013683] LNet: Removed LNI 192.168.202.1@tcp [ 894.023850] LNet: Removed LNI 192.168.202.1@tcp2 [ 894.046276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.122798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 894.154708] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 894.162308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.168036] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 901.623837] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:01:36 (1788620496) [ 901.921225] LNet: 45973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.931818] LNetError: 45973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.936705] LNetError: 45973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 901.947771] LNet: Removed LNI 10.1.2.3@tcp [ 901.949494] LNet: Removed LNI 192.168.202.1@tcp [ 901.960386] LNet: Removed LNI 192.168.202.1@tcp2 [ 902.732873] Key type .llcrypt unregistered [ 902.737934] Key type ._llcrypt unregistered [ 909.589543] Key type ._llcrypt registered [ 909.592440] Key type .llcrypt registered [ 909.815606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 909.830439] alg: No test for adler32 (adler32-zlib) [ 910.739817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.800763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.821806] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 911.697978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.716386] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.723266] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.751432] LNet: Removed LNI 192.168.202.1@tcp [ 911.772177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.903572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 911.940892] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 912.486334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.504541] LNet: 46395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.509633] LNetError: 46395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.527404] LNet: Removed LNI 192.168.202.1@tcp [ 912.546973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.600980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.624561] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 913.033524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.076716] LNet: Removed LNI 192.168.202.1@tcp [ 913.098498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.169955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.199788] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 913.647582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.765832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.768555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.772712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.161447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.193667] LNet: 46504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.196772] LNet: 46504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 916.209069] LNetError: 46504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.214403] LNetError: 46504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 916.228966] LNet: Removed LNI 192.168.202.1@tcp [ 916.259148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.323581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.346949] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 921.491498] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:01:56 (1788620516) [ 921.607708] LNet: 47094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.626181] LNetError: 47094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.652240] LNet: Removed LNI 192.168.202.1@tcp [ 922.432812] Key type .llcrypt unregistered [ 922.434346] Key type ._llcrypt unregistered [ 928.373963] Key type ._llcrypt registered [ 928.375895] Key type .llcrypt registered [ 928.578447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.590548] alg: No test for adler32 (adler32-zlib) [ 929.639032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.729254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.784396] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 931.497622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.514921] LNet: 47496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.521388] LNetError: 47496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.534889] LNet: Removed LNI 192.168.202.1@tcp [ 931.550155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.663607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 931.696481] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 938.793707] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:02:14 (1788620534) [ 938.857357] LNet: 48081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.860246] LNetError: 48081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.867869] LNet: Removed LNI 192.168.202.1@tcp [ 939.317526] Key type .llcrypt unregistered [ 939.319064] Key type ._llcrypt unregistered [ 945.146427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.415737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.419363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.423448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.961381] Key type ._llcrypt registered [ 947.966966] Key type .llcrypt registered [ 948.205781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.224024] alg: No test for adler32 (adler32-zlib) [ 949.172742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.216488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.248236] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 950.162336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.185364] LNet: 48537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.194660] LNetError: 48537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.213359] LNet: Removed LNI 192.168.202.1@tcp [ 950.246353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.310078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 950.335918] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 956.603587] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:02:31 (1788620551) [ 956.857909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.917360] LNet: 49135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.925885] LNetError: 49135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.938338] LNet: Removed LNI 192.168.202.1@tcp [ 956.956284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.035331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 962.125907] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:02:37 (1788620557) [ 962.213321] LNet: 49707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.800751] Key type .llcrypt unregistered [ 962.802980] Key type ._llcrypt unregistered [ 967.684537] Key type ._llcrypt registered [ 967.687828] Key type .llcrypt registered [ 967.906108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.925522] alg: No test for adler32 (adler32-zlib) [ 968.860946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.908266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.923501] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 969.635683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.659117] LNet: 50111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.665459] LNetError: 50111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.692579] LNet: Removed LNI 192.168.202.1@tcp [ 969.705516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.772953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 969.809159] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 975.154465] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:02:50 (1788620570) [ 975.289370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.315338] LNet: 50703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.319820] LNetError: 50703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.333312] LNet: Removed LNI 192.168.202.1@tcp [ 975.347767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.401910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 981.057516] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:02:56 (1788620576) [ 982.317895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 983.647102] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:02:59 (1788620579) [ 985.154389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 986.628400] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:03:01 (1788620581) [ 987.739655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 989.388915] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:03:04 (1788620584) [ 990.765776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 992.655169] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:03:07 (1788620587) [ 994.092321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 995.879532] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:03:11 (1788620591) [ 996.113777] LNet: 52756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.664656] Key type .llcrypt unregistered [ 996.666652] Key type ._llcrypt unregistered [ 1002.792324] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1002.912988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1002.938911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1002.952119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1005.605641] Key type ._llcrypt registered [ 1005.607920] Key type .llcrypt registered [ 1005.789103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1005.807370] alg: No test for adler32 (adler32-zlib) [ 1006.737947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.804364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.829441] LNet: Added LNI 192.168.202.1@tcp [8/512/0/180] [ 1006.836975] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1006.841449] LNet: 53075:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.936462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.955708] LNet: 53094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.960749] LNetError: 53094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.963186] LNet: Removed LNI 10.1.2.3@tcp [ 1006.968294] LNetError: 53094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.984165] LNet: Removed LNI 192.168.202.1@tcp [ 1006.999821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.055960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.074912] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1007.126143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.141995] LNet: 53118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.147943] LNetError: 53118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.159253] LNet: Removed LNI 192.168.202.1@tcp [ 1007.177502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.247253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.266532] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1007.270395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.272581] LNet: 53126:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.351171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.371643] LNet: 53144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.375284] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.380309] LNet: Removed LNI 10.1.2.3@tcp [ 1007.382398] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.390154] LNet: Removed LNI 192.168.202.1@tcp [ 1007.411354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.461176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.479109] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1007.481957] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.485565] LNet: 53152:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.623176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.645739] LNet: 53178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.648931] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.654216] LNet: Removed LNI 10.1.2.3@tcp [ 1007.658264] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.670533] LNet: Removed LNI 192.168.202.1@tcp [ 1007.708614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.758892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.775741] LNet: Added LNI 192.168.202.1@tcp [8/256/0/182] [ 1007.780107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1007.866968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.884455] LNet: 53206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.895387] LNetError: 53206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.901127] LNet: Removed LNI 10.1.2.3@tcp [ 1007.902696] LNetError: 53206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.914155] LNet: Removed LNI 192.168.202.1@tcp [ 1007.931605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.992555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1008.008953] LNet: Added LNI 192.168.202.1@tcp [8/256/0/182] [ 1008.014498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1008.017227] LNet: 53214:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1008.083790] LNet: Removed LNI 10.1.2.3@tcp [ 1008.088914] LNet: Removed LNI 192.168.202.1@tcp [ 1008.555172] Key type .llcrypt unregistered [ 1008.557306] Key type ._llcrypt unregistered [ 1017.871786] Key type ._llcrypt registered [ 1017.873844] Key type .llcrypt registered [ 1019.384408] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:03:34 (1788620614) [ 1020.893636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1022.670750] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:03:37 (1788620617) [ 1022.958097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.964850] alg: No test for adler32 (adler32-zlib) [ 1024.021753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.111229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.153806] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1024.819846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.1 [ 1024.840866] LNet: 54475:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1024.852673] LNet: Removed LNI 192.168.202.1@tcp [ 1031.153413] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:03:46 (1788620626) [ 1031.243088] LNet: 55038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.254776] LNetError: 55038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.288285] LNet: Removed LNI 192.168.202.1@tcp [ 1031.967638] Key type .llcrypt unregistered [ 1031.973078] Key type ._llcrypt unregistered [ 1039.558186] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1039.896897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1042.825183] Key type ._llcrypt registered [ 1042.828257] Key type .llcrypt registered [ 1043.100643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.126470] alg: No test for adler32 (adler32-zlib) [ 1044.548521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1051.684964] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:04:06 (1788620646) [ 1051.861547] LNet: 55947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.874936] LNetError: 55947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1051.896662] LNet: Removed LNI 10.1.2.3@tcp [ 1052.677440] Key type .llcrypt unregistered [ 1052.680278] Key type ._llcrypt unregistered [ 1059.604356] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1059.861372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1059.871355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1059.878638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1062.706257] Key type ._llcrypt registered [ 1062.714836] Key type .llcrypt registered [ 1062.949308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.970784] alg: No test for adler32 (adler32-zlib) [ 1064.001879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1070.358658] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:04:25 (1788620665) [ 1071.809213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1073.272146] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:04:28 (1788620668) [ 1073.550644] LNet: 57140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.559233] LNetError: 57140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1073.584258] LNet: Removed LNI 10.1.2.3@tcp [ 1074.043626] Key type .llcrypt unregistered [ 1074.046213] Key type ._llcrypt unregistered [ 1079.806241] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1080.075376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1080.086214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1080.099104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1082.867213] Key type ._llcrypt registered [ 1082.869132] Key type .llcrypt registered [ 1083.029577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.046563] alg: No test for adler32 (adler32-zlib) [ 1084.086419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.177220] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1092.052396] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:04:47 (1788620687) [ 1092.391497] LNet: 58365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.397483] LNetError: 58365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.430423] LNet: Removed LNI 10.1.2.3@tcp [ 1092.911691] Key type .llcrypt unregistered [ 1092.914153] Key type ._llcrypt unregistered [ 1098.621542] Key type ._llcrypt registered [ 1098.635436] Key type .llcrypt registered [ 1098.930288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1098.949619] alg: No test for adler32 (adler32-zlib) [ 1099.808946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1099.824682] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1107.212904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1107.797756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_interrupt -r 1 [ 1107.855753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_interrupt -r 1 [ 1107.913764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1107.955592] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1108.064636] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1108.222689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.253934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.290543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.317151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.346110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1108.390643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.421237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.455798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.492691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.524787] LNet: 58818:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.548578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.592198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.624961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.656702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.688739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.714651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.738518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.268459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_dropped -r 1 [ 1109.296879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_dropped -r 1 [ 1109.327401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1109.348329] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1109.354890] LNet: Skipped 1 previous similar message [ 1109.551564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.574886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.601624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.625744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.645764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1109.663392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.696348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.720669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.756195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.773955] LNet: 58929:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.791382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.825275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.863170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.890025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.927541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.953752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.985096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.478823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_aborted -r 1 [ 1110.498330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_aborted -r 1 [ 1110.521721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1110.538508] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1110.743896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.772847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.819635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.853616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.908066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.939913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.981642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.007390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.032594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.065324] LNet: 59041:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.094335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.123413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.145971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.171082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.203928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.241883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.280661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.793674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_no_route -r 1 [ 1111.828696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_no_route -r 1 [ 1111.863110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1112.068667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.094068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.129672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.147965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.170511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1112.205458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.222740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.246110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.260833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.273559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.297629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.315701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.347387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.401224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.426831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.486713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1112.972606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_timeout -r 1 [ 1113.004157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_timeout -r 1 [ 1113.036390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1113.050318] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1113.054795] LNet: Skipped 2 previous similar messages [ 1113.224409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.249128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.280248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.311147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.337476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1113.376816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.408558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.452627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.484493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.530163] LNet: 59263:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.540428] LNet: 59263:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1113.566493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.594395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.625436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.655935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.691831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.715211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.743278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.235276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_error -r 1 [ 1114.261849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_error -r 1 [ 1114.287152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1114.552279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.579484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.606900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.637064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.668928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1114.719825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.762255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.822696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.845415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.868878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.930883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.976219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.014581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.050313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.078939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.119291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.341846] LNet: 59415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.348655] LNetError: 59415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.419128] LNet: Removed LNI 192.168.202.1@tcp [ 1116.883763] Key type .llcrypt unregistered [ 1116.890189] Key type ._llcrypt unregistered [ 1135.380398] Key type ._llcrypt registered [ 1135.383129] Key type .llcrypt registered [ 1136.847248] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:05:31 (1788620731) [ 1137.704526] Key type .llcrypt unregistered [ 1137.707110] Key type ._llcrypt unregistered [ 1144.619326] Key type ._llcrypt registered [ 1144.624170] Key type .llcrypt registered [ 1144.858928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.876734] alg: No test for adler32 (adler32-zlib) [ 1145.847273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.866987] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1153.317284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1164.031639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.077799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1164.104408] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1165.507892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_interrupt -r 1 [ 1165.536818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_interrupt -r 1 [ 1165.568376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_interrupt -r 1 [ 1165.602659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_interrupt -r 1 [ 1165.642800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_interrupt -r 1 [ 1165.695327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_interrupt -r 1 [ 1165.732732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_interrupt -r 1 [ 1165.765995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_interrupt -r 1 [ 1165.805785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1165.826536] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1165.833144] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1166.053980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.103709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.155326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.204113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.242648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1166.294212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.340333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.368650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.422405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.453139] LNet: 60880:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.479916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.538445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.571522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.619687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.678174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.717845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.786811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.335279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_dropped -r 1 [ 1167.368667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_dropped -r 1 [ 1167.393855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_dropped -r 1 [ 1167.425346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_dropped -r 1 [ 1167.449172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_dropped -r 1 [ 1167.477210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_dropped -r 1 [ 1167.532453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_dropped -r 1 [ 1167.602606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_dropped -r 1 [ 1167.658711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1167.682947] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1167.694309] LNet: Skipped 2 previous similar messages [ 1167.900822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.925204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.953633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.979874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.006444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1168.037395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.066929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.103657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.135995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.153479] LNet: 61003:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.179686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.212442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.240572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.276973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.327726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.366834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.459490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.964534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_aborted -r 1 [ 1168.998854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_aborted -r 1 [ 1169.023730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_aborted -r 1 [ 1169.070362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_aborted -r 1 [ 1169.107853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_aborted -r 1 [ 1169.151985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_aborted -r 1 [ 1169.200923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_aborted -r 1 [ 1169.252720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_aborted -r 1 [ 1169.307160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1169.338159] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1169.344280] LNet: Skipped 2 previous similar messages [ 1169.600059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.623478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.646444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.675491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.704910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1169.732655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.776456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.818315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.866843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.891280] LNet: 61126:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.921713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.969279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.010858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.037577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.063599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.089719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.119147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.613771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_no_route -r 1 [ 1170.652708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_no_route -r 1 [ 1170.693443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_no_route -r 1 [ 1170.724757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_no_route -r 1 [ 1170.796726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_no_route -r 1 [ 1170.864571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_no_route -r 1 [ 1170.897330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_no_route -r 1 [ 1170.928235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_no_route -r 1 [ 1170.966305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1171.264545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.313324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.375780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.431416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.496944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1171.548499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.589271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.643825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.683304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.716472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.749948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.784117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.821948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.851557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.905874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.949406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.564640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_timeout -r 1 [ 1172.606985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_timeout -r 1 [ 1172.667143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_timeout -r 1 [ 1172.707688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_timeout -r 1 [ 1172.734739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_timeout -r 1 [ 1172.756518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_timeout -r 1 [ 1172.787477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_timeout -r 1 [ 1172.821423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_timeout -r 1 [ 1172.878695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1172.922374] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1172.937455] LNet: Skipped 5 previous similar messages [ 1173.145665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.206613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.303296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.342945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.372913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.397719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.452796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.514150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.550700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.567432] LNet: 61373:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.574426] LNet: 61373:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1173.603823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.659137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.759669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.822081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.886910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.953922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.030384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.738171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_error -r 1 [ 1174.766469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_error -r 1 [ 1174.820266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_error -r 1 [ 1174.871471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_error -r 1 [ 1174.926416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_error -r 1 [ 1174.953482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_error -r 1 [ 1174.986996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_error -r 1 [ 1175.024529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_error -r 1 [ 1175.056610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1175.288569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.363985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.402722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.437062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.467980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1175.514828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.557650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.586551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.631320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.677784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.715831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.752446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.775856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.803599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.835979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.883485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.097203] LNet: 61572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.112597] LNetError: 61572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.126871] LNet: Removed LNI 192.168.202.1@tcp [ 1183.145893] LNet: Removed LNI 192.168.202.1@tcp1 [ 1183.824076] Key type .llcrypt unregistered [ 1183.827106] Key type ._llcrypt unregistered [ 1200.540362] Key type ._llcrypt registered [ 1200.542378] Key type .llcrypt registered [ 1201.611869] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:06:36 (1788620796) [ 1202.276418] Key type .llcrypt unregistered [ 1202.279540] Key type ._llcrypt unregistered [ 1208.445305] Key type ._llcrypt registered [ 1208.449201] Key type .llcrypt registered [ 1208.610185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.622369] alg: No test for adler32 (adler32-zlib) [ 1209.572153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1209.628858] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1217.311807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1217.874931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_dropped -r 1 [ 1217.911122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_dropped -r 1 [ 1217.954575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1217.981078] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1218.164419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.182458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.198632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.227180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.253157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1218.276847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.334726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.358591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.380180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.396518] LNet: 62818:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.423695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.453106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.474525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.497742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.513785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.532083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.548938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1218.917615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_error -r 1 [ 1218.942930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_error -r 1 [ 1218.971945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1218.989578] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1219.100242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.113295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.126258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.144876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.160252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1219.174831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.188851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.202618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.220709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.230579] LNet: 62929:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.240205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.262643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.286184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.307922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.339058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.395802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.438766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1219.766331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_timeout -r 1 [ 1219.789652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_timeout -r 1 [ 1219.807149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1219.940509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.953641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.968715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.985644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.003510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1220.026632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.064793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.088779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.109372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.133842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.159797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.183108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.212390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1220.245400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.278750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.312680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.640232] LNet: 63081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.650909] LNetError: 63081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.669064] LNet: Removed LNI 192.168.202.1@tcp [ 1221.065410] Key type .llcrypt unregistered [ 1221.067606] Key type ._llcrypt unregistered [ 1235.483294] Key type ._llcrypt registered [ 1235.485987] Key type .llcrypt registered [ 1236.459443] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:07:11 (1788620831) [ 1237.327296] Key type .llcrypt unregistered [ 1237.328467] Key type ._llcrypt unregistered [ 1243.181100] Key type ._llcrypt registered [ 1243.187366] Key type .llcrypt registered [ 1243.316882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1243.324856] alg: No test for adler32 (adler32-zlib) [ 1244.268470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1244.304966] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1251.692925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1261.717377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.781180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1261.847148] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1263.891035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_dropped -r 1 [ 1263.942912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e remote_dropped -r 1 [ 1264.001058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_dropped -r 1 [ 1264.055206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e remote_dropped -r 1 [ 1264.121104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e remote_dropped -r 1 [ 1264.172644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e remote_dropped -r 1 [ 1264.223643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e remote_dropped -r 1 [ 1264.299571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e remote_dropped -r 1 [ 1264.346331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1264.370759] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1264.377818] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.101@tcp [ 1264.621429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.656401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.727609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.768273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.827521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1264.887527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.927362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.956963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.991559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.015242] LNet: 64544:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.047233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.085952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.137637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.171084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.207535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.244527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.305995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1265.359497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.042631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_error -r 1 [ 1266.086501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e remote_error -r 1 [ 1266.144766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_error -r 1 [ 1266.183796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e remote_error -r 1 [ 1266.230167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e remote_error -r 1 [ 1266.322787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e remote_error -r 1 [ 1266.438816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e remote_error -r 1 [ 1266.511929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e remote_error -r 1 [ 1266.595133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1266.639519] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1266.643179] LNet: Skipped 2 previous similar messages [ 1266.951619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.983381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.027730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.103240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.171467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1267.232322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.306590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.362963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.406816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.437495] LNet: 64669:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1267.506243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.594334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.675940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.722408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.776218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.832947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.910055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.981826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.801637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_timeout -r 1 [ 1268.856739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e remote_timeout -r 1 [ 1268.913613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_timeout -r 1 [ 1268.959737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e remote_timeout -r 1 [ 1269.043881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e remote_timeout -r 1 [ 1269.130401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e remote_timeout -r 1 [ 1269.203261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e remote_timeout -r 1 [ 1269.257227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e remote_timeout -r 1 [ 1269.338709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.101@tcp [ 1269.366116] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1269.370925] LNet: Skipped 1 previous similar message [ 1269.707546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.742313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1269.790778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.849198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.883360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1269.919158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.960342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.994876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.028969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.043710] LNet: 64795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.060208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.085903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.101767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.119565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.137817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.157425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.180765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.200781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.877239] LNet: 64872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.892007] LNetError: 64872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.904651] LNet: Removed LNI 192.168.202.1@tcp [ 1274.942148] LNet: Removed LNI 192.168.202.1@tcp1 [ 1275.451934] Key type .llcrypt unregistered [ 1275.457894] Key type ._llcrypt unregistered [ 1291.464713] Key type ._llcrypt registered [ 1291.465889] Key type .llcrypt registered [ 1292.498500] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:08:07 (1788620887) [ 1293.792726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1295.361345] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:08:10 (1788620890) [ 1295.920463] Key type .llcrypt unregistered [ 1295.922449] Key type ._llcrypt unregistered [ 1301.548536] Key type ._llcrypt registered [ 1301.551593] Key type .llcrypt registered [ 1301.748163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1301.764984] alg: No test for adler32 (adler32-zlib) [ 1302.635364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1302.657203] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1310.358913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1311.231468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e network_timeout -r 1 [ 1311.279556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e network_timeout -r 1 [ 1311.311784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1311.333262] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1311.618284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.640804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1311.666770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.718969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.756773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1311.805910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.857841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.904551] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1311.922885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.979972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.009175] LNet: 66411:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.037225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.095317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.150879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.200097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.251342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.289348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.574772] LNet: 66450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.582095] LNetError: 66450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.594875] LNet: Removed LNI 192.168.202.1@tcp [ 1312.998797] Key type .llcrypt unregistered [ 1313.000776] Key type ._llcrypt unregistered [ 1327.671968] Key type ._llcrypt registered [ 1327.674370] Key type .llcrypt registered [ 1328.043460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.064212] alg: No test for adler32 (adler32-zlib) [ 1329.122374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.174508] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1337.765761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1346.781504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.807719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.824713] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1348.779700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e network_timeout -r 1 [ 1348.807521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e network_timeout -r 1 [ 1348.839971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e network_timeout -r 1 [ 1348.873163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e network_timeout -r 1 [ 1348.933872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e network_timeout -r 1 [ 1348.960934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e network_timeout -r 1 [ 1348.994556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e network_timeout -r 1 [ 1349.024766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e network_timeout -r 1 [ 1349.042111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1349.054572] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1349.260599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1349.292155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1349.331319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1349.358945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1349.381526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1349.411320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.436419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.459703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.480198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.491355] LNet: 67326:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.503168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.522190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.542112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.559020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.573189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.589343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.643428] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1354.784411] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1354.871606] LNet: 67400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.874543] LNetError: 67400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.880172] LNet: Removed LNI 192.168.202.1@tcp [ 1354.901402] LNet: Removed LNI 192.168.202.1@tcp1 [ 1355.282784] Key type .llcrypt unregistered [ 1355.287170] Key type ._llcrypt unregistered [ 1376.038257] Key type ._llcrypt registered [ 1376.042383] Key type .llcrypt registered [ 1377.677032] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:09:32 (1788620972) [ 1378.337844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.364159] alg: No test for adler32 (adler32-zlib) [ 1379.437475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.570707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.601565] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1380.363259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1380.389048] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1381.469627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 1381.561440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1381.654564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1381.705190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1381.775122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1381.855519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1381.917655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.1@tcp [ 1381.962486] LNetError: 68499:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.1@tcp to recovery [ 1382.559318] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1382.561416] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1384.607547] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1385.564955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1388.706531] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1389.295174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1389.362467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1389.438956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.491695] LNet: 68606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.502410] LNetError: 68606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.510270] LNet: Removed LNI 192.168.202.1@tcp [ 1389.517983] LNet: Removed LNI 192.168.202.1@tcp1 [ 1389.548614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.640737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.661276] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1390.125889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.138920] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1390.842813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 1390.946861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.052674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1391.101724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1391.146524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1391.178338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1391.204175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1391.252361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.1@tcp [ 1391.268665] LNetError: 68878:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.1@tcp to recovery [ 1391.647390] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1394.813398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1397.791855] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1397.800693] LNet: Skipped 1 previous similar message [ 1402.910914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1402.941346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1402.969089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.436742] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:10:04 (1788621004) [ 1409.531838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.571414] LNet: 69590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.579792] LNetError: 69590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.585357] LNetError: 69590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1409.590579] LNet: Removed LNI 192.168.202.1@tcp [ 1409.614863] LNet: Removed LNI 192.168.202.1@tcp1 [ 1409.677718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.744467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.772798] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1410.452252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1410.475995] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1411.300224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 1411.358781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1411.384811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1411.413665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1411.463277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1411.509699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1411.563575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.1@tcp --health 0 [ 1411.668661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1411.743226] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.1@tcp [ 1412.767420] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1412.774063] LNet: Skipped 1 previous similar message [ 1426.135878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.261169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1426.301335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.328526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.1@tcp --health 500 [ 1426.435541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.180591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.272428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.289396] LNet: 70078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.293861] LNetError: 70078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.302304] LNetError: 70078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.310524] LNet: Removed LNI 192.168.202.1@tcp [ 1433.323351] LNet: Removed LNI 192.168.202.1@tcp1 [ 1433.342368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.422514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1433.459258] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1434.328744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1434.358733] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1435.373264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 1435.401836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1435.457782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1435.487886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1435.516842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1435.560142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1435.584739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1435.609343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.1@tcp --health 0 [ 1436.448194] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1436.457273] LNet: Skipped 3 previous similar messages [ 1436.838510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1446.871673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1446.912738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1446.941111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.287825] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:10:48 (1788621048) [ 1454.001397] LNet: 71124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.006958] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.010833] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1454.014930] LNet: Removed LNI 192.168.202.1@tcp [ 1454.023936] LNet: Removed LNI 192.168.202.1@tcp1 [ 1454.580730] Key type .llcrypt unregistered [ 1454.581822] Key type ._llcrypt unregistered [ 1461.508900] Key type ._llcrypt registered [ 1461.516285] Key type .llcrypt registered [ 1461.859772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1461.888796] alg: No test for adler32 (adler32-zlib) [ 1462.880476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1462.898031] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1472.776792] Lustre: DEBUG MARKER: Initial discovery [ 1472.832861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.101@tcp [ 1475.027976] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1475.245273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1475.297192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.101@tcp [ 1475.326230] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1475.349786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1475.385810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.101@tcp -r 1 -m GET -l 3 [ 1475.419429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.101@tcp -r 1 -m GET -e local_error [ 1475.440498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.101@tcp -r 1 -m PUT -l 6 [ 1476.192052] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1476.597552] Lustre: DEBUG MARKER: Force oleg201-client.virtnet to discover 192.168.202.101@tcp (in background) [ 1476.764740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.101@tcp [ 1477.851690] Lustre: DEBUG MARKER: Force oleg201-server to discover 192.168.202.1@tcp [ 1479.852278] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1480.296294] Lustre: DEBUG MARKER: Wait for 71884 [ 1481.787774] Lustre: DEBUG MARKER: Finished wait on 71884 [ 1481.964887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1481.987246] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1481.996412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1482.284861] LNet: 72333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.294698] LNetError: 72333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.320768] LNet: Removed LNI 192.168.202.1@tcp [ 1482.785688] Key type .llcrypt unregistered [ 1482.790254] Key type ._llcrypt unregistered [ 1500.577193] Key type ._llcrypt registered [ 1500.579449] Key type .llcrypt registered [ 1501.769258] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:11:36 (1788621096) [ 1502.287609] Key type .llcrypt unregistered [ 1502.291423] Key type ._llcrypt unregistered [ 1508.254794] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1508.383780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1508.389608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1508.393838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1511.080402] Key type ._llcrypt registered [ 1511.083315] Key type .llcrypt registered [ 1511.301866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1511.321755] alg: No test for adler32 (adler32-zlib) [ 1512.217701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.285555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.330610] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1512.994721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1513.010597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1520.553696] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:11:55 (1788621115) [ 1520.747041] LNet: 74396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.772182] LNetError: 74396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1520.789499] LNet: Removed LNI 10.1.2.3@tcp [ 1520.817077] LNet: Removed LNI 192.168.202.1@tcp [ 1521.391488] Key type .llcrypt unregistered [ 1521.393159] Key type ._llcrypt unregistered [ 1526.936996] Key type ._llcrypt registered [ 1526.938706] Key type .llcrypt registered [ 1527.169871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.181285] alg: No test for adler32 (adler32-zlib) [ 1528.085422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.129745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.166965] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1529.000663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1529.016309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1529.887864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.1@tcp [ 1530.977530] LNetError: Unexpected error -22 connecting to 192.168.202.1@tcp at host 192.168.202.1:988 [ 1535.191444] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:12:10 (1788621130) [ 1535.256795] LNet: 75518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.267699] LNetError: 75518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1535.274872] LNet: Removed LNI 10.1.2.3@tcp [ 1535.292632] LNet: Removed LNI 192.168.202.1@tcp [ 1535.640876] Key type .llcrypt unregistered [ 1535.643952] Key type ._llcrypt unregistered [ 1540.670525] Key type ._llcrypt registered [ 1540.673429] Key type .llcrypt registered [ 1540.948456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.975436] alg: No test for adler32 (adler32-zlib) [ 1541.839795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.884975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1541.908925] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1542.572867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1542.592803] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 1543.419658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.1@tcp1 --nid 192.168.202.1@tcp2 [ 1548.878461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp1 192.168.202.1@tcp2 [ 1548.905568] LNetError: 76359:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.1@tcp1 to 192.168.202.1@tcp2 - Route table may be misconfigured [ 1548.928454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp2 192.168.202.1@tcp1 [ 1556.868218] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:12:31 (1788621151) [ 1557.068463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.088962] LNet: 76932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.094804] LNetError: 76932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.111329] LNet: Removed LNI 192.168.202.1@tcp1 [ 1558.115410] LNetError: 76932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.172213] LNet: Removed LNI 192.168.202.1@tcp2 [ 1558.260615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.377116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.413712] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1559.044862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1559.063525] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1559.846902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 1559.913061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.1@tcp [ 1559.986118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp1 192.168.202.1@tcp1 [ 1560.034127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.1@tcp -d 192.168.202.1@tcp -e network_timeout [ 1560.094305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -e network_timeout [ 1560.142946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -e network_timeout [ 1560.199098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -e network_timeout [ 1560.257605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1560.296893] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1560.322606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1560.416923] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1560.493855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.606459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1567.208749] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:12:42 (1788621162) [ 1567.676616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.689781] LNet: 77793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.693950] LNetError: 77793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.736390] LNet: Removed LNI 192.168.202.1@tcp [ 1569.766897] LNet: Removed LNI 192.168.202.1@tcp1 [ 1569.787134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.826296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1570.055270] LNet: 77831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.552647] Key type .llcrypt unregistered [ 1570.554878] Key type ._llcrypt unregistered [ 1579.495606] Key type ._llcrypt registered [ 1579.498698] Key type .llcrypt registered [ 1580.642329] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:12:55 (1788621175) [ 1580.950695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1580.957810] alg: No test for adler32 (adler32-zlib) [ 1581.921235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.290079] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1584.679738] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:13:00 (1788621180) [ 1585.017260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.039430] LNet: 78926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.067425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.126794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.147916] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1585.914515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1585.936733] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1586.802567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1586.839289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp1 [ 1586.895671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp1 [ 1592.927425] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:13:08 (1788621188) [ 1594.603301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg201-server [ 1596.113312] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:13:11 (1788621191) [ 1597.877650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg201-server [ 1599.561529] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:13:14 (1788621194) [ 1600.972225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg201-server [ 1602.314702] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:13:17 (1788621197) [ 1603.992436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg201-server [ 1605.512307] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:13:20 (1788621200) [ 1607.142553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg201-server [ 1608.691891] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:13:23 (1788621203) [ 1610.339716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg201-server [ 1611.796744] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:13:27 (1788621207) [ 1613.327463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg201-server [ 1614.706664] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:13:30 (1788621210) [ 1616.237716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg201-server [ 1617.737187] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:13:33 (1788621213) [ 1619.486485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg201-server [ 1620.927379] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:13:36 (1788621216) [ 1621.025639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.062757] LNet: 82798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.068596] LNetError: 82798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.075872] LNet: Removed LNI 192.168.202.1@tcp [ 1622.120269] LNet: Removed LNI 192.168.202.1@tcp1 [ 1622.168847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.215575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.247852] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1622.832948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1622.918693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1623.056487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.076204] LNet: 82949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.081886] LNetError: 82949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.086568] LNetError: 82949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1624.106667] LNet: Removed LNI 192.168.202.1@tcp [ 1624.145715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.203890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1624.241525] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1624.952658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1625.131534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.150808] LNet: 83094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.155613] LNetError: 83094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.171220] LNet: Removed LNI 192.168.202.1@tcp [ 1625.180781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.214388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.231214] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1625.955993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1626.034744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1626.191327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.238188] LNet: Removed LNI 192.168.202.1@tcp [ 1627.265843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.328198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1627.361825] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1628.108297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1628.313225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.329820] LNetError: 83387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.335347] LNetError: 83387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1628.349666] LNet: Removed LNI 192.168.202.1@tcp [ 1628.373171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.433638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.454044] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1629.241184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1629.324410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1629.532709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.552568] LNet: 83534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.558691] LNet: 83534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1630.629973] LNet: Removed LNI 192.168.202.1@tcp [ 1630.662043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.714524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1630.764821] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1631.535414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1631.767699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.873256] LNet: Removed LNI 192.168.202.1@tcp [ 1632.915480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.967775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.996067] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1633.537926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1633.602323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1633.754885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.769312] LNetError: 83826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.774311] LNetError: 83826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1633.781461] LNet: Removed LNI 192.168.202.1@tcp [ 1633.796314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.838753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1633.872701] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1634.772826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1635.010574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.081083] LNet: Removed LNI 192.168.202.1@tcp [ 1636.121149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.241487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.273833] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1636.992686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1637.093449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1637.406605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.502274] LNet: Removed LNI 192.168.202.1@tcp [ 1638.576900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.707050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1638.754268] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1639.597162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1639.925230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.953876] LNet: 84264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.958396] LNet: 84264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1641.012465] LNet: Removed LNI 192.168.202.1@tcp [ 1641.048526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.107988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.127350] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1641.999877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1642.085293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1642.379926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.446719] LNetError: 84412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.460418] LNetError: 84412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1643.494043] LNet: Removed LNI 192.168.202.1@tcp [ 1643.549404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.621540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1643.651401] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1644.398685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1645.872589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.955363] LNet: Removed LNI 192.168.202.1@tcp [ 1647.007954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.102173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.141648] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1647.881664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1647.924291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.942595] LNet: Removed LNI 192.168.202.1@tcp [ 1647.954352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.988115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.011806] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1648.702640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1648.757572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.775503] LNet: Removed LNI 192.168.202.1@tcp [ 1648.784671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.820073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1648.883553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.910809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.947694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1649.005696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.032752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.079074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1649.136481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.177588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.233964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1649.252465] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1649.802735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.826365] LNet: Removed LNI 192.168.202.1@tcp [ 1649.844054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.909613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1655.553470] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:14:10 (1788621250) [ 1655.719057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.776666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.842907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1655.865788] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/180] [ 1657.184508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.201391] LNet: 85735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.205867] LNet: 85735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1657.222986] LNet: Removed LNI 192.168.202.1@tcp231 [ 1657.240187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.301102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1657.323988] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/180] [ 1657.328178] LNet: 85743:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.679885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.715095] LNet: Removed LNI 192.168.202.1@tcp231 [ 1657.746155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1657.789364] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/180] [ 1658.879467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.900541] LNetError: 85887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.905297] LNetError: 85887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1658.918710] LNet: Removed LNI 192.168.202.1@tcp231 [ 1658.932959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.991390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1659.007081] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/0] [ 1659.011784] LNet: 85895:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.453857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1659.477499] LNet: Removed LNI 192.168.202.1@tcp231 [ 1659.493279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1659.528248] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/0] [ 1660.628441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.672322] LNet: Removed LNI 192.168.202.1@tcp231 [ 1660.699215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.764254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1660.798269] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/60] [ 1660.808083] LNet: 86046:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.173869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1661.198059] LNet: Removed LNI 192.168.202.1@tcp231 [ 1661.216600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1661.242137] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/60] [ 1662.409207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.444065] LNet: Removed LNI 192.168.202.1@tcp231 [ 1662.458739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.542422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1662.587305] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/180] [ 1663.088879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1663.111769] LNet: Removed LNI 192.168.202.1@tcp231 [ 1663.134423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1663.181787] LNet: Added LNI 192.168.202.1@tcp231 [8/256/0/180] [ 1671.116442] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:14:26 (1788621266) [ 1671.237720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.322950] LNet: Removed LNI 192.168.202.1@tcp231 [ 1671.373734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.448666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1671.477912] LNet: Added LNI 192.168.202.1@tcp232 [8/256/0/180] [ 1672.425489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1672.475367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1672.535202] LNet: Removed LNI 192.168.202.1@tcp232 [ 1672.572495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1672.615897] LNet: Added LNI 192.168.202.1@tcp232 [8/256/0/180] [ 1673.398861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1673.419793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1673.446547] LNet: Removed LNI 192.168.202.1@tcp232 [ 1673.464781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1673.492729] LNet: Added LNI 192.168.202.1@tcp232 [8/256/0/180] [ 1674.397621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1674.467403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1674.530036] LNet: Removed LNI 192.168.202.1@tcp232 [ 1674.563945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1674.622264] LNet: Added LNI 192.168.202.1@tcp232 [8/256/0/180] [ 1675.486692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1675.545384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1675.610068] LNet: Removed LNI 192.168.202.1@tcp232 [ 1675.633103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1680.790128] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:14:36 (1788621276) [ 1681.629493] Key type .llcrypt unregistered [ 1681.631352] Key type ._llcrypt unregistered [ 1687.174937] Key type ._llcrypt registered [ 1687.183194] Key type .llcrypt registered [ 1687.707686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1687.755368] alg: No test for adler32 (adler32-zlib) [ 1688.809297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1688.859447] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1696.896488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1709.623843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.689579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1709.706432] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1711.119407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_interrupt -r 2 [ 1711.169292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_interrupt -r 2 [ 1711.199853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_interrupt -r 2 [ 1711.234300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_interrupt -r 2 [ 1711.271826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_interrupt -r 2 [ 1711.300313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_interrupt -r 2 [ 1711.324085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_interrupt -r 2 [ 1711.361290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_interrupt -r 2 [ 1711.391613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1711.409574] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1711.417734] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1711.640402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1711.663545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1711.691321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1711.714225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1711.748376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1711.773915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1711.803401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1711.836693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1711.861881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1711.876755] LNet: 88699:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1711.893754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1711.922743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1711.961274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1712.022410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1712.052990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1712.076258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1712.732657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_dropped -r 2 [ 1712.762786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_dropped -r 2 [ 1712.792254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_dropped -r 2 [ 1712.823318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_dropped -r 2 [ 1712.859359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_dropped -r 2 [ 1712.894209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_dropped -r 2 [ 1712.931375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_dropped -r 2 [ 1712.968514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_dropped -r 2 [ 1713.020235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1713.043397] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1713.335397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1713.361361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1713.385802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1713.414626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1713.437537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1713.462892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1713.486894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1713.517647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1713.551338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1713.571625] LNet: 88822:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1713.595663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1713.641995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1713.679400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1713.729502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1713.760686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1713.788867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1714.403669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_aborted -r 2 [ 1714.455142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_aborted -r 2 [ 1714.512837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_aborted -r 2 [ 1714.550497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_aborted -r 2 [ 1714.591583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_aborted -r 2 [ 1714.635244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_aborted -r 2 [ 1714.680741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_aborted -r 2 [ 1714.751581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_aborted -r 2 [ 1714.824556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1714.850518] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1715.167052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.202538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1715.236754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.273479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1715.318532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1715.375604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1715.407366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1715.463142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1715.507854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1715.523459] LNet: 88944:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.539121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1715.567838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1715.591798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1715.615141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1715.658923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1715.690104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.071770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_no_route -r 2 [ 1716.097643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_no_route -r 2 [ 1716.130193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_no_route -r 2 [ 1716.159756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_no_route -r 2 [ 1716.189668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_no_route -r 2 [ 1716.218246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_no_route -r 2 [ 1716.287330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_no_route -r 2 [ 1716.361090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_no_route -r 2 [ 1716.412346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1716.746585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.774564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1716.803251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.825482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.874265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1716.910642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.953465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.019341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.061989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1717.103284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1717.161025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1717.211228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1717.280755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1717.364264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1717.404766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.973630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e local_timeout -r 2 [ 1718.002608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e local_timeout -r 2 [ 1718.031169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e local_timeout -r 2 [ 1718.061970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e local_timeout -r 2 [ 1718.097240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e local_timeout -r 2 [ 1718.124211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e local_timeout -r 2 [ 1718.146652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e local_timeout -r 2 [ 1718.178912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e local_timeout -r 2 [ 1718.211974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1718.248678] LNet: There was an unexpected network error while writing to 192.168.202.101: rc = -22 [ 1718.251464] LNet: Skipped 1 previous similar message [ 1718.470413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.502476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.536284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.567242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.617591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1718.680131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.732290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.822736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.909771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.937682] LNet: 89189:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.945876] LNet: 89189:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1718.974508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.016589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.079666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.119514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.170651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.223343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.781543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp -m GET -e remote_dropped -r 2 [ 1719.828336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.101@tcp1 -m GET -e remote_dropped -r 2 [ 1719.870789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp -m GET -e remote_dropped -r 2 [ 1719.918929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp -d 192.168.202.1@tcp1 -m GET -e remote_dropped -r 2 [ 1719.972827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp -m GET -e remote_dropped -r 2 [ 1720.015215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.1@tcp1 -d 192.168.202.101@tcp1 -m GET -e remote_dropped -r 2 [ 1720.041645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp -m GET -e remote_dropped -r 2 [ 1720.076476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.1@tcp1 -d 192.168.202.1@tcp1 -m GET -e remote_dropped -r 2 [ 1720.110520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.1@tcp 192.168.202.101@tcp [ 1720.135170] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.101@tcp [ 1720.394602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.450415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.512155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.548732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.584469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1720.608088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.625635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.651556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.678573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.703153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.740648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.784295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.813187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.848708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.904521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1726.904591] LNet: 89385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.913614] LNetError: 89385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.919244] LNet: Removed LNI 192.168.202.1@tcp [ 1726.927891] LNet: Removed LNI 192.168.202.1@tcp1 [ 1727.445444] Key type .llcrypt unregistered [ 1727.447137] Key type ._llcrypt unregistered [ 1744.734523] Key type ._llcrypt registered [ 1744.737147] Key type .llcrypt registered [ 1746.010673] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:15:41 (1788621341) [ 1746.389815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.400425] alg: No test for adler32 (adler32-zlib) [ 1747.339943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.386459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.404641] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1748.159284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1748.241706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.1@tcp [ 1748.296373] LNetError: 90360:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.1@tcp to recovery [ 1748.386228] LNet: 1 local NIs in recovery (showing 1): 192.168.202.1@tcp [ 1748.457723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.1@tcp [ 1748.624248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.260989] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:15:49 (1788621349) [ 1754.492603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.524920] LNet: 90950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.531521] LNetError: 90950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.565369] LNet: Removed LNI 192.168.202.1@tcp [ 1755.597607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.686033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.726988] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1756.469907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1756.486200] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 1757.230259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.1@tcp [ 1757.247349] LNetError: 91217:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.1@tcp to recovery [ 1757.665706] LNet: There was an unexpected network error while writing to 192.168.202.1: rc = -22 [ 1758.538228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.1@tcp [ 1766.412482] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:16:01 (1788621361) [ 1768.312893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg201-server [ 1770.178967] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:16:05 (1788621365) [ 1771.625126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg201-server [ 1773.287543] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:16:08 (1788621368) [ 1774.848947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg201-server [ 1776.277966] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:16:11 (1788621371) [ 1776.406593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.427169] LNet: 92841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.434041] LNetError: 92841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.439167] LNet: Removed LNI 192.168.202.1@tcp [ 1777.440674] LNetError: 92841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.465356] LNet: Removed LNI 192.168.202.1@tcp1 [ 1777.532359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.628766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.651709] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1778.543559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1778.624225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1778.672739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1778.776768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1778.818947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1778.943513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1785.468831] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:16:20 (1788621380) [ 1785.621691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.677979] LNet: 93576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.691393] LNetError: 93576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.737431] LNet: Removed LNI 192.168.202.1@tcp [ 1785.757410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.800575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.815084] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1792.984111] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:16:28 (1788621388) [ 1793.123470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.165284] LNet: 94275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.176323] LNetError: 94275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.204267] LNet: Removed LNI 192.168.202.1@tcp [ 1793.265915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.309759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.323899] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1800.367563] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:16:35 (1788621395) [ 1801.707479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1803.781820] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:16:38 (1788621398) [ 1804.253619] LNet: 95317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.263785] LNetError: 95317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.285859] LNet: Removed LNI 192.168.202.1@tcp [ 1804.875711] Key type .llcrypt unregistered [ 1804.879314] Key type ._llcrypt unregistered [ 1811.735189] Key type ._llcrypt registered [ 1811.737221] Key type .llcrypt registered [ 1811.952747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.979775] alg: No test for adler32 (adler32-zlib) [ 1812.931696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1812.962667] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1821.110968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 1827.502979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.101@tcp [ 1828.879411] LNet: 95731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.888635] LNetError: 95731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.900696] LNet: Removed LNI 192.168.202.1@tcp [ 1829.217464] Key type .llcrypt unregistered [ 1829.219560] Key type ._llcrypt unregistered [ 1840.933831] Key type ._llcrypt registered [ 1840.935695] Key type .llcrypt registered [ 1842.065220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1843.603677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1845.279792] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:17:20 (1788621440) [ 1846.775625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1848.587971] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:17:23 (1788621443) [ 1850.348997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg201-server [ 1852.261207] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:17:27 (1788621447) [ 1853.948799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg201-server [ 1855.916313] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:17:30 (1788621450) [ 1857.661536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.714113] alg: No test for adler32 (adler32-zlib) [ 1858.770991] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1859.081199] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1860.831300] Key type lgssc registered [ 1862.124029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.496407] Key type lgssc unregistered [ 1883.908740] LNet: 98902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.923714] LNetError: 98902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.955778] LNet: Removed LNI 192.168.202.1@tcp [ 1885.328513] Key type .llcrypt unregistered [ 1885.330939] Key type ._llcrypt unregistered [ 1901.987648] Key type ._llcrypt registered [ 1901.989813] Key type .llcrypt registered [ 1903.428259] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:18:18 (1788621498) [ 1903.947265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.962991] alg: No test for adler32 (adler32-zlib) [ 1904.986374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.044403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1905.088825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1905.144338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1905.191722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1911.389304] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:18:26 (1788621506) [ 1911.510644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.540296] LNet: 100310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.564509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.621865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.652414] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1912.841393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.894782] LNet: 100460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.904212] LNetError: 100460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.927279] LNet: Removed LNI 192.168.202.1@tcp [ 1912.951700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.009539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1913.039549] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1913.538273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.563951] LNetError: 100488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.584255] LNet: Removed LNI 192.168.202.1@tcp [ 1913.612293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.671714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1913.697683] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1920.275066] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:18:35 (1788621515) [ 1920.426917] LNet: 101073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.439918] LNet: 101073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1920.453401] LNetError: 101073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.487534] LNet: Removed LNI 192.168.202.1@tcp [ 1920.917567] Key type .llcrypt unregistered [ 1920.922328] Key type ._llcrypt unregistered [ 1926.562786] Key type ._llcrypt registered [ 1926.565547] Key type .llcrypt registered [ 1926.851267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1926.874983] alg: No test for adler32 (adler32-zlib) [ 1927.859851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.909915] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1927.912493] LNet: *** cfs_fail_loc=e003, val=100*** [ 1928.701466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1928.722639] LNet: *** cfs_fail_loc=e003, val=100*** [ 1928.726546] LNet: Skipped 2 previous similar messages [ 1928.737348] Ping reply received with mapping: [ 1928.741480] found metadata reply NI[192.168.202.1@tcp] [ 1928.745856] 01 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1928.748960] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1928.753699] 5a 5a 5a 5a 5a 5a 5a 5a [ 1928.923295] LNet: 101473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.936249] LNetError: 101473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.964568] LNet: Removed LNI 192.168.202.1@tcp [ 1929.741483] Key type .llcrypt unregistered [ 1929.743329] Key type ._llcrypt unregistered [ 1941.970864] Key type ._llcrypt registered [ 1941.973322] Key type .llcrypt registered [ 1943.717543] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:18:58 (1788621538) [ 1944.271506] Key type .llcrypt unregistered [ 1944.273972] Key type ._llcrypt unregistered [ 1951.752305] Key type ._llcrypt registered [ 1951.753960] Key type .llcrypt registered [ 1951.942835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1951.963239] alg: No test for adler32 (adler32-zlib) [ 1953.024886] LNetError: 102510:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1953.370289] Key type .llcrypt unregistered [ 1953.371427] Key type ._llcrypt unregistered [ 1958.787921] Key type ._llcrypt registered [ 1958.790205] Key type .llcrypt registered [ 1960.291229] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:19:15 (1788621555) [ 1960.765402] Key type .llcrypt unregistered [ 1960.768277] Key type ._llcrypt unregistered [ 1967.194538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1967.202821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1969.755054] Key type ._llcrypt registered [ 1969.759370] Key type .llcrypt registered [ 1969.957704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1969.985371] alg: No test for adler32 (adler32-zlib) [ 1970.985747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.067293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1980.763605] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:19:36 (1788621576) [ 1980.956996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.974326] LNet: 104285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.978577] LNetError: 104285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1980.989569] LNet: Removed LNI 10.1.2.3@tcp [ 1980.999622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.744833] Key type .llcrypt unregistered [ 1981.747407] Key type ._llcrypt unregistered [ 1989.413532] Key type ._llcrypt registered [ 1989.415818] Key type .llcrypt registered [ 1989.658753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1989.674754] alg: No test for adler32 (adler32-zlib) [ 1990.694723] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 1990.793321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 1991.012610] LNet: 104571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.025468] LNetError: 104571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.053318] LNet: Removed LNI 192.168.202.1@tcp [ 1991.931541] Key type .llcrypt unregistered [ 1991.935438] Key type ._llcrypt unregistered [ 2004.042103] Key type ._llcrypt registered [ 2004.043995] Key type .llcrypt registered [ 2004.976077] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:20:00 (1788621600) [ 2005.558323] Key type .llcrypt unregistered [ 2005.559353] Key type ._llcrypt unregistered [ 2013.243320] Key type ._llcrypt registered [ 2013.245624] Key type .llcrypt registered [ 2013.504987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.552241] alg: No test for adler32 (adler32-zlib) [ 2023.284410] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:20:18 (1788621618) [ 2023.557880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.583514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.637758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2023.661340] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2024.236514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.283852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2024.341565] LNet: Removed LNI 192.168.202.1@tcp [ 2024.614333] LNet: 106311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.207383] Key type .llcrypt unregistered [ 2025.216877] Key type ._llcrypt unregistered [ 2038.935673] Key type ._llcrypt registered [ 2038.937609] Key type .llcrypt registered [ 2040.549663] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:20:35 (1788621635) [ 2041.094099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.129093] alg: No test for adler32 (adler32-zlib) [ 2042.278763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.396516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.442331] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2043.101308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2043.155550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.1@tcp [ 2043.182729] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2043.189609] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.205114] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2043, 0, 0 [ 2043.220403] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2043.227419] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2043.234518] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.239513] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2043, 0, 0 [ 2043.258623] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2043.261615] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2043.266367] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.269514] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2043, 0, 2093 [ 2043.274527] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2043.277418] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2043.281250] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2043.283432] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2043, 0, 2093 [ 2043.289302] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2050.851156] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:20:45 (1788621645) [ 2051.403124] LNet: 107849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.420832] LNetError: 107849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.468174] LNet: Removed LNI 192.168.202.1@tcp [ 2052.250655] Key type .llcrypt unregistered [ 2052.269380] Key type ._llcrypt unregistered [ 2058.953553] Key type ._llcrypt registered [ 2058.954774] Key type .llcrypt registered [ 2059.066613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2059.094684] alg: No test for adler32 (adler32-zlib) [ 2059.995809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2060.025308] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2068.267154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 2077.797395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.882966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2077.924402] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 2078.865132] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2079.064435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2079.073115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2079.089820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2081.508544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2081.550891] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2093.009592] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:21:28 (1788621688) [ 2093.359423] LNet: 109358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.366857] LNetError: 109358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.372745] LNet: Removed LNI 192.168.202.1@tcp [ 2094.435675] LNet: Removed LNI 192.168.202.1@tcp1 [ 2094.438476] LNetError: 109358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2094.452560] LNetError: 109358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2094.488792] LNet: Removed LNI 10.1.2.3@tcp99 [ 2095.180858] Key type .llcrypt unregistered [ 2095.183513] Key type ._llcrypt unregistered [ 2102.073470] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2102.077065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2104.609467] Key type ._llcrypt registered [ 2104.613072] Key type .llcrypt registered [ 2104.885593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2104.904507] alg: No test for adler32 (adler32-zlib) [ 2105.818327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.858540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.873790] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2106.826513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2106.862895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2109.329445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2109.369869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2109.413765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2109.436320] LNetError: 110099:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2109.468670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2109.505416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2110.157119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2110.255417] LNet: 110119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.266376] LNetError: 110119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2110.271895] LNet: Removed LNI 10.1.2.3@tcp [ 2111.337166] LNet: Removed LNI 192.168.202.1@tcp [ 2112.174453] Key type .llcrypt unregistered [ 2112.177289] Key type ._llcrypt unregistered [ 2127.101198] Key type ._llcrypt registered [ 2127.111466] Key type .llcrypt registered [ 2128.586449] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:22:03 (1788621723) [ 2128.875571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2128.886861] alg: No test for adler32 (adler32-zlib) [ 2129.778121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.804860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.823224] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2130.589508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2130.672671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg201-client@tcp [ 2136.286555] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:22:11 (1788621731) [ 2136.469146] LNet: 111650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.480586] LNetError: 111650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.512345] LNet: Removed LNI 192.168.202.1@tcp [ 2138.049559] Key type .llcrypt unregistered [ 2138.051620] Key type ._llcrypt unregistered [ 2144.419619] Key type ._llcrypt registered [ 2144.424810] Key type .llcrypt registered [ 2144.607277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.623207] alg: No test for adler32 (adler32-zlib) [ 2145.534386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2145.594482] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2145.645667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2152.459849] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:22:27 (1788621747) [ 2153.813531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2155.421645] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:22:30 (1788621750) [ 2156.852197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2158.444696] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:22:33 (1788621753) [ 2158.604490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.631722] LNet: 113068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.643089] LNetError: 113068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.669142] LNet: Removed LNI 192.168.202.1@tcp [ 2158.697039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.742553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.771469] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2159.274742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2159.328950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2165.626482] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:22:40 (1788621760) [ 2165.730431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.754648] LNet: 113774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.759207] LNetError: 113774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.770688] LNet: Removed LNI 192.168.202.1@tcp [ 2165.792707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.871133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2165.911428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2166.352477] Key type .llcrypt unregistered [ 2166.355090] Key type ._llcrypt unregistered [ 2180.649078] Key type ._llcrypt registered [ 2180.650461] Key type .llcrypt registered [ 2182.283499] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:22:57 (1788621777) [ 2182.825365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.852700] alg: No test for adler32 (adler32-zlib) [ 2183.807952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.874665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.904390] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2184.975332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2185.042533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2192.725692] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:23:07 (1788621787) [ 2192.860366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.882216] LNet: 115331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.892108] LNetError: 115331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.909435] LNet: Removed LNI 192.168.202.1@tcp [ 2192.941382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.991473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2193.033473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2200.679447] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:23:15 (1788621795) [ 2200.759021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.771824] LNet: 115914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.792882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.845661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2201.034201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2201.057050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2201.107593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2201.315901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2201.350507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2201.418452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.448404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.497153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2201.839323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2201.867987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2201.950807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2202.247859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2202.271559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2209.152310] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:23:23 (1788621803) [ 2209.414861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.439225] LNet: 116547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.446420] LNet: 116547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2209.468794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.538257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.557602] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2210.126330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2210.138860] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2210.734308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2210.777399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2210.938376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.949896] LNetError: 116826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.960296] LNet: Removed LNI 192.168.202.1@tcp [ 2210.975828] LNet: Removed LNI 192.168.202.1@tcp2 [ 2210.995138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.080171] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2211.090730] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2211.100796] LNet: 116832:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.855611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2212.111557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.136126] LNet: 116869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.140581] LNet: 116869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2212.153337] LNetError: 116869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.156747] LNetError: 116869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2212.170210] LNet: Removed LNI 192.168.202.1@tcp [ 2212.182312] LNet: Removed LNI 192.168.202.1@tcp2 [ 2212.200182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.265973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.284481] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2213.173294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.209784] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2214.245590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2214.305162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2214.403630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2214.716875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.757898] LNetError: 117154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.768800] LNetError: 117154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2214.785197] LNet: Removed LNI 192.168.202.1@tcp [ 2215.845582] LNet: Removed LNI 192.168.202.1@tcp2 [ 2215.910216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.000573] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2216.010314] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2216.021880] LNet: 117160:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2216.693587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2216.918844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.935414] LNet: 117197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.939251] LNet: 117197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2216.943703] LNet: Removed LNI 192.168.202.1@tcp [ 2216.956256] LNet: Removed LNI 192.168.202.1@tcp2 [ 2216.974737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.011788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2217.029574] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2217.781852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2217.802629] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2218.820525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2218.874748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2219.113689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.126285] LNetError: 117476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.130598] LNetError: 117476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2219.135953] LNet: Removed LNI 192.168.202.1@tcp [ 2219.151829] LNet: Removed LNI 192.168.202.1@tcp2 [ 2219.196746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.279202] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2219.283809] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2219.289555] LNet: 117482:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.149689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2220.489536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.599407] LNet: Removed LNI 192.168.202.1@tcp [ 2221.610844] LNet: Removed LNI 192.168.202.1@tcp2 [ 2221.690610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.798294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.859120] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2222.614544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2222.639499] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2223.857058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2223.961195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2224.072496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2224.425757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.503263] LNet: Removed LNI 192.168.202.1@tcp [ 2226.533809] LNet: Removed LNI 192.168.202.1@tcp2 [ 2226.596430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.703789] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2226.712740] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2226.720405] LNet: 117810:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.340432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2227.711597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.771628] LNet: 117848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.783860] LNet: 117848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2227.797571] LNetError: 117848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.811949] LNetError: 117848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2227.822213] LNet: Removed LNI 192.168.202.1@tcp [ 2228.838685] LNet: Removed LNI 192.168.202.1@tcp2 [ 2228.856729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.932296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.964891] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2229.711946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2229.743310] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2230.449079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2230.475821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2230.538737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2231.018119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.095169] LNet: Removed LNI 192.168.202.1@tcp [ 2232.108869] LNet: Removed LNI 192.168.202.1@tcp2 [ 2232.138123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.255102] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2232.259597] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2232.264204] LNet: 118151:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2232.780799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2233.288524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.336469] LNet: Removed LNI 192.168.202.1@tcp [ 2234.350443] LNet: Removed LNI 192.168.202.1@tcp2 [ 2234.377138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.452140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.497672] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2235.276278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2235.295000] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2236.283127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2236.350958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2236.429741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2236.484578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2236.889603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.984125] LNet: Removed LNI 192.168.202.1@tcp [ 2239.014581] LNet: Removed LNI 192.168.202.1@tcp2 [ 2239.044502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.095127] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2239.102740] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2239.625047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2245.859354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2247.449548] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:24:02 (1788621842) [ 2247.777780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.794325] LNet: 119270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.798328] LNet: 119270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2247.803424] LNetError: 119270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.807454] LNetError: 119270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2247.812940] LNet: Removed LNI 192.168.202.1@tcp [ 2248.898418] LNet: Removed LNI 192.168.202.1@tcp2 [ 2248.934306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.198991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2249.228488] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2249.996624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2250.016445] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.988506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.010781] LNet: Removed LNI 10.1.2.3@tcp [ 2251.018860] LNet: Removed LNI 192.168.202.1@tcp [ 2251.044600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.130237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2251.180900] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2251.186047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.754962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2251.781837] LNet: Removed LNI 192.168.202.1@tcp [ 2251.796647] LNet: Removed LNI 10.1.2.3@tcp [ 2258.499555] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:24:13 (1788621853) [ 2258.684423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.724577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.760269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.782558] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2259.633735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2259.654671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2259.684370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.714443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.752664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.794869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.829432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.856724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.884897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.914513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2259.943451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2260.026296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2260.159177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2266.115496] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 11:24:21 (1788621861) [ 2266.255379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.303541] LNet: Removed LNI 192.168.202.1@tcp [ 2267.331049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.409221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.425174] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2268.024526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2268.052976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2268.093324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.132419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.164801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.203744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.241841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.281152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.352190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.403551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.439371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.474467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.547733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2268.591944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.625449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.657975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.703382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.762626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.802435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.834770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.878123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2268.934936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2269.014649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.1@tcp [ 2276.935592] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:24:31 (1788621871) [ 2277.043542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2277.112321] LNet: Removed LNI 192.168.202.1@tcp [ 2277.174454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.331543] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2277.577857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2277.595620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2277.601790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2279.938681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2280.010699] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2280.948746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2280.993810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2283.612643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2283.652565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2283.713348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.1@tcp -m PUT -e local_timeout -r 1 [ 2283.779562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2283.874951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2283.888112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2284.228701] LNetError: 121706:0:(lib-move.c:4794:lnet_parse()) 192.168.202.1@tcp, src 192.168.202.1@tcp: Dropping PUT (error -108 looking up sender) [ 2285.279409] LNet: 122165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.281428] LNet: 122165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2285.284823] LNetError: 122165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2285.287155] LNetError: 122165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2285.289549] LNet: Removed LNI 10.1.2.3@tcp [ 2286.312516] LNet: Removed LNI 192.168.202.1@tcp [ 2287.137110] Key type .llcrypt unregistered [ 2287.140965] Key type ._llcrypt unregistered [ 2302.449239] Key type ._llcrypt registered [ 2302.452197] Key type .llcrypt registered [ 2303.965346] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:24:58 (1788621898) [ 2304.464645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.477189] alg: No test for adler32 (adler32-zlib) [ 2305.564586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.327478] LNet: 123016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.468475] Key type .llcrypt unregistered [ 2307.477555] Key type ._llcrypt unregistered [ 2316.004425] Key type ._llcrypt registered [ 2316.012338] Key type .llcrypt registered [ 2316.301778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2316.333361] alg: No test for adler32 (adler32-zlib) [ 2317.297861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2317.345258] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2325.177864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 2325.401565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2326.077073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.101@tcp [ 2326.274910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2326.302887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2326.340874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2326.384798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2326.409923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2326.436534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2326.461894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2326.487527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2326.509946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2326.532322] LNet: 123465:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2326.559182] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.101@tcp [ 2326.572860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2326.616809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2326.644885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2326.671778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2326.724754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2326.760562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2327.557490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2327.801611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2327.849843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2327.901352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2327.939944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2328.035667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2328.081971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2328.142171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2328.218446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2328.262435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2328.296544] LNet: 123584:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2328.336141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2328.395884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2328.442450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2328.482972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2328.522418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2328.575117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2329.520547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.101@tcp [ 2329.782578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2329.807733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2329.826434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2329.843223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2329.863163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2329.884798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2329.914637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2329.936205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2329.952323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2329.961083] LNet: 123703:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2329.969122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2329.983453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2329.997986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2330.015217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2330.036224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2330.051210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2330.850628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2331.044935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2331.081881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2331.103362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2331.135939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2331.186453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2331.215563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2331.254376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2331.282472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2331.307961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2331.334474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2331.353721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2331.375890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2331.395210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2331.415982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2331.433851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2331.623598] LNet: 123869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.626543] LNetError: 123869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.638638] LNet: Removed LNI 192.168.202.1@tcp [ 2331.860866] Key type .llcrypt unregistered [ 2331.861942] Key type ._llcrypt unregistered [ 2355.532023] Key type ._llcrypt registered [ 2355.533882] Key type .llcrypt registered [ 2357.797755] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:25:51 (1788621951) [ 2358.430177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.466288] alg: No test for adler32 (adler32-zlib) [ 2359.628264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.436140] LNet: 124754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.429348] Key type .llcrypt unregistered [ 2361.432987] Key type ._llcrypt unregistered [ 2371.508810] Key type ._llcrypt registered [ 2371.510763] Key type .llcrypt registered [ 2371.647393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2371.656375] alg: No test for adler32 (adler32-zlib) [ 2372.687954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2372.716734] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2379.912904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.101@tcp [ 2389.757996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.823142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2389.865698] LNet: Added LNI 192.168.202.1@tcp1 [8/256/0/180] [ 2390.691996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2391.387327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.101@tcp [ 2391.674041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2391.716387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2391.778848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2391.814735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2391.853823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2391.884311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2391.928889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2391.964271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2392.002346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2392.033165] LNet: 125406:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2392.047727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2392.104161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2392.141981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2392.179664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2392.215623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2392.223299] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.101@tcp [ 2392.271528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2392.904442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2393.102448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2393.133992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2393.159408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2393.194931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2393.216203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2393.238657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2393.256567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2393.302258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2393.350691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2393.384873] LNet: 125525:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.408725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2393.493332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2393.550311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2393.598726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2393.620685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2393.645834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2394.206548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.101@tcp [ 2394.522302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2394.572608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2394.607269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2394.634748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2394.662626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2394.697519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2394.746150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2394.792306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2394.837768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2394.863853] LNet: 125644:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2394.886935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2394.929402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2394.970546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2395.001577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2395.038893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2395.079589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2395.643493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2395.861294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2395.891115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2395.917227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2395.943721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2395.966982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2395.986684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2396.018627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2396.050768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2396.086461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2396.116794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2396.148646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2396.174510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2396.205112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2396.233836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2396.276096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2401.056914] LNet: 125844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.069234] LNetError: 125844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.077877] LNet: Removed LNI 192.168.202.1@tcp [ 2401.096070] LNet: Removed LNI 192.168.202.1@tcp1 [ 2401.567747] Key type .llcrypt unregistered [ 2401.570276] Key type ._llcrypt unregistered [ 2417.641774] Key type ._llcrypt registered [ 2417.643946] Key type .llcrypt registered [ 2418.651902] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:26:54 (1788622014) [ 2419.223102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.232985] alg: No test for adler32 (adler32-zlib) [ 2420.195761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.270414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.314475] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2421.018478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2421.055199] LNet: Added LNI 192.168.202.1@tcp2 [8/256/0/180] [ 2422.140780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.1@tcp [ 2422.237965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2423.263162] LNet: Removed LNI 192.168.202.1@tcp [ 2430.282736] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:27:05 (1788622025) [ 2431.817378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2433.696860] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:27:08 (1788622028) [ 2440.695460] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:27:16 (1788622036) [ 2447.929596] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:27:23 (1788622043) [ 2454.183318] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:27:29 (1788622049) [ 2461.334281] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:27:36 (1788622056) [ 2469.867595] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:27:44 (1788622064) [ 2478.603493] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:27:53 (1788622073) [ 2485.881327] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:28:01 (1788622081) [ 2492.427560] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:28:07 (1788622087) [ 2500.797355] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:28:15 (1788622095) [ 2508.483601] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:28:23 (1788622103) [ 2516.449631] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:28:31 (1788622111) [ 2523.284236] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:28:38 (1788622118) [ 2530.636149] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:28:45 (1788622125) [ 2539.249503] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:28:54 (1788622134) [ 2547.872923] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:29:03 (1788622143) [ 2556.918793] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:29:11 (1788622151) [ 2565.149198] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:29:19 (1788622159) [ 2565.272838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2565.322837] LNet: 137529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.329282] LNetError: 137529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.346064] LNet: Removed LNI 192.168.202.1@tcp2 [ 2565.379631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.567542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2565.586225] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2567.150396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2567.180400] LNet: 137678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2567.185235] LNetError: 137678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2567.202459] LNet: Removed LNI 192.168.202.1@tcp [ 2567.229906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2567.314603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2567.347130] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2575.223394] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:29:30 (1788622170) [ 2583.116218] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:29:38 (1788622178) [ 2591.449483] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:29:46 (1788622186) [ 2599.253968] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:29:54 (1788622194) [ 2608.042021] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:30:02 (1788622202) [ 2616.768969] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:30:11 (1788622211) [ 2625.368377] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:30:20 (1788622220) [ 2633.774262] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:30:28 (1788622228) [ 2642.899558] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:30:37 (1788622237) [ 2653.639765] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:30:48 (1788622248) [ 2661.665339] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:30:56 (1788622256) [ 2669.342541] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:31:04 (1788622264) [ 2675.802193] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2397 sec ========== 11:31:10 (1788622270) [ 2676.096830] LNet: 145145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.104921] LNetError: 145145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.122917] LNet: Removed LNI 192.168.202.1@tcp [ 2676.640569] Key type .llcrypt unregistered [ 2676.646224] Key type ._llcrypt unregistered [ 2688.994588] Key type ._llcrypt registered [ 2688.997930] Key type .llcrypt registered [ 2689.818936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2689.877566] alg: No test for adler32 (adler32-zlib) [ 2691.185862] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 2691.473470] LNet: Added LNI 192.168.202.1@tcp [8/256/0/180] [ 2693.288914] Key type lgssc registered [ 2695.222052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.855424] Lustre: Mounted lustre-client - version 2.17.58_3_g10061d6 [ 2773.009660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2779.820573] Lustre: Unmounted lustre-client [ 2798.350728] Key type lgssc unregistered [ 2798.641852] LNet: 148094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.648194] LNetError: 148094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.673813] LNet: Removed LNI 192.168.202.1@tcp [ 2799.257305] Key type .llcrypt unregistered [ 2799.260762] Key type ._llcrypt unregistered