[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436141454 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002442] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007716] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010298] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.022654] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026778] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029012] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.032218] *** VALIDATE proc *** [ 0.033221] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.036030] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043363] debug: unmapping init [mem 0xffffffff87659000-0xffffffff87660fff] [ 0.045253] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049024] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.055229] rcu: Hierarchical SRCU implementation. [ 0.057405] smp: Bringing up secondary CPUs ... [ 0.058600] x86: Booting SMP configuration: [ 0.059019] .... node #0, CPUs: #1 #2 #3 [ 0.064020] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.203342] node 0 deferred pages initialised in 132ms [ 0.207153] devtmpfs: initialized [ 0.209407] x86/mm: Memory block size: 128MB [ 0.213074] gcov: version magic: 0x41383552 [ 0.216354] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.224438] pinctrl core: initialized pinctrl subsystem [ 0.226343] [ 0.226934] ************************************************************* [ 0.229020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232020] ** ** [ 0.235017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.237015] ** ** [ 0.241017] ** This means that this kernel is built to expose internal ** [ 0.243016] ** IOMMU data structures, which may compromise security on ** [ 0.246019] ** your system. ** [ 0.249029] ** ** [ 0.252017] ** If you see this message and you are not debugging the ** [ 0.254023] ** kernel, report this immediately to your vendor! ** [ 0.257018] ** ** [ 0.260016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263019] ************************************************************* [ 0.266890] NET: Registered protocol family 16 [ 0.269585] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.273976] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.283037] cpuidle: using governor menu [ 0.285122] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.287805] PCI: Using configuration type 1 for base access [ 0.290153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301152] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.304042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308087] cryptd: max_cpu_qlen set to 1000 [ 0.313279] ACPI: Added _OSI(Module Device) [ 0.315030] ACPI: Added _OSI(Processor Device) [ 0.316015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322886] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329516] ACPI: Interpreter enabled [ 0.331078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333017] ACPI: Using IOAPIC for interrupt routing [ 0.335147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354080] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.361097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366489] acpiphp: Slot [2] registered [ 0.367152] acpiphp: Slot [5] registered [ 0.370246] acpiphp: Slot [6] registered [ 0.373205] acpiphp: Slot [3] registered [ 0.374278] acpiphp: Slot [4] registered [ 0.377152] acpiphp: Slot [7] registered [ 0.379221] acpiphp: Slot [8] registered [ 0.381190] acpiphp: Slot [9] registered [ 0.382097] acpiphp: Slot [10] registered [ 0.383125] acpiphp: Slot [11] registered [ 0.385142] acpiphp: Slot [12] registered [ 0.387141] acpiphp: Slot [13] registered [ 0.389134] acpiphp: Slot [14] registered [ 0.390106] acpiphp: Slot [15] registered [ 0.392147] acpiphp: Slot [16] registered [ 0.394171] acpiphp: Slot [17] registered [ 0.396150] acpiphp: Slot [18] registered [ 0.398138] acpiphp: Slot [19] registered [ 0.399145] acpiphp: Slot [20] registered [ 0.401103] acpiphp: Slot [21] registered [ 0.403112] acpiphp: Slot [22] registered [ 0.404111] acpiphp: Slot [23] registered [ 0.405123] acpiphp: Slot [24] registered [ 0.406112] acpiphp: Slot [25] registered [ 0.408143] acpiphp: Slot [26] registered [ 0.409116] acpiphp: Slot [27] registered [ 0.410102] acpiphp: Slot [28] registered [ 0.412092] acpiphp: Slot [29] registered [ 0.413102] acpiphp: Slot [30] registered [ 0.415127] acpiphp: Slot [31] registered [ 0.416095] PCI host bridge to bus 0000:00 [ 0.417022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.421028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.423027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.426030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.431964] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435509] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.444020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.448128] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.453022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.455023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.460634] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.463781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.467065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.470439] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.473906] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.482021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.487017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.491949] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.502985] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.519019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528530] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.540014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563041] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.566691] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.569470] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.572551] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.574274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.580033] iommu: Default domain type: Passthrough [ 0.581405] SCSI subsystem initialized [ 0.583127] ACPI: bus type USB registered [ 0.585090] usbcore: registered new interface driver usbfs [ 0.586081] usbcore: registered new interface driver hub [ 0.588069] usbcore: registered new device driver usb [ 0.590159] pps_core: LinuxPPS API ver. 1 registered [ 0.592010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.595053] PTP clock support registered [ 0.597165] EDAC MC: Ver: 3.0.0 [ 0.599533] PCI: Using ACPI for IRQ routing [ 0.601854] NetLabel: Initializing [ 0.603015] NetLabel: domain hash size = 128 [ 0.605016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.607113] NetLabel: unlabeled traffic allowed by default [ 0.610121] vgaarb: loaded [ 0.612345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.614020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.620718] clocksource: Switched to clocksource kvm-clock [ 0.732353] VFS: Disk quotas dquot_6.6.0 [ 0.733963] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.736648] *** VALIDATE ramfs *** [ 0.737732] *** VALIDATE hugetlbfs *** [ 0.739251] pnp: PnP ACPI init [ 0.741718] pnp: PnP ACPI: found 6 devices [ 0.760187] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.763472] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.765583] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.767269] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.769757] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772352] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.774997] NET: Registered protocol family 2 [ 0.777726] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.782931] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.786500] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.791269] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.794460] TCP: Hash tables configured (established 65536 bind 65536) [ 0.797322] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.799456] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801757] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803649] NET: Registered protocol family 1 [ 0.805679] RPC: Registered named UNIX socket transport module. [ 0.807457] RPC: Registered udp transport module. [ 0.808773] RPC: Registered tcp transport module. [ 0.810616] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.812544] NET: Registered protocol family 44 [ 0.813762] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.815275] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.817022] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.819409] PCI: CLS 0 bytes, default 64 [ 0.820946] Unpacking initramfs... [ 2.210449] debug: unmapping init [mem 0xffff941d7cc64000-0xffff941d7ffcffff] [ 2.213895] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.215942] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.218453] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.717593] Initialise system trusted keyrings [ 2.719293] Key type blacklist registered [ 2.721421] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.734239] zbud: loaded [ 2.737837] *** VALIDATE nfs *** [ 2.739350] *** VALIDATE nfs4 *** [ 2.741524] pstore: using deflate compression [ 2.745264] Platform Keyring initialized [ 2.851281] NET: Registered protocol family 38 [ 2.853017] Key type asymmetric registered [ 2.854746] Asymmetric key parser 'x509' registered [ 2.856894] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.860093] io scheduler mq-deadline registered [ 2.862149] io scheduler kyber registered [ 2.864177] io scheduler bfq registered [ 2.866870] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.870237] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.874408] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.877430] ACPI: Power Button [PWRF] [ 2.882933] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.889701] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.905681] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.932815] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.962720] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.967694] Non-volatile memory driver v1.3 [ 2.969560] Linux agpgart interface v0.103 [ 2.999640] virtio_blk virtio1: [vda] 146664 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.002973] vda: detected capacity change from 0 to 75091968 [ 3.020626] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.024104] vdb: detected capacity change from 0 to 1073741824 [ 3.033577] libphy: Fixed MDIO Bus: probed [ 3.040350] usbcore: registered new interface driver usbserial_generic [ 3.043297] usbserial: USB Serial support registered for generic [ 3.045857] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.050249] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.052463] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.055169] mousedev: PS/2 mouse device common for all mice [ 3.057618] rtc_cmos 00:05: RTC can wake from S4 [ 3.060344] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.062161] rtc_cmos 00:05: registered as rtc0 [ 3.067414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.071648] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.071983] intel_pstate: CPU model not supported [ 3.079423] hid: raw HID events driver (C) Jiri Kosina [ 3.079688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.081535] usbcore: registered new interface driver usbhid [ 3.087086] usbhid: USB HID core driver [ 3.088936] drop_monitor: Initializing network drop monitor service [ 3.092780] Initializing XFRM netlink socket [ 3.095270] NET: Registered protocol family 10 [ 3.099723] Segment Routing with IPv6 [ 3.101098] NET: Registered protocol family 17 [ 3.102926] mpls_gso: MPLS GSO support [ 3.108776] RAS: Correctable Errors collector initialized. [ 3.112380] AVX version of gcm_enc/dec engaged. [ 3.115309] AES CTR mode by8 optimization enabled [ 3.195127] sched_clock: Marking stable (3195078627, 0)->(4081627535, -886548908) [ 3.199152] registered taskstats version 1 [ 3.201202] Loading compiled-in X.509 certificates [ 3.203742] zswap: loaded using pool lzo/zbud [ 3.229345] Key type big_key registered [ 3.241681] Key type encrypted registered [ 3.243115] ima: No TPM chip found, activating TPM-bypass! [ 3.245061] ima: Allocated hash algorithm: sha1 [ 3.246699] ima: No architecture policies found [ 3.249903] evm: Initialising EVM extended attributes: [ 3.253336] evm: security.selinux [ 3.254642] evm: security.ima [ 3.255738] evm: security.capability [ 3.257191] evm: HMAC attrs: 0x1 [ 3.259586] rtc_cmos 00:05: setting system clock to 2026-09-01 11:41:47 UTC (1788262907) [ 3.266769] debug: unmapping init [mem 0xffffffff88603000-0xffffffff887fffff] [ 3.269816] debug: unmapping init [mem 0xffffffff87382000-0xffffffff87658fff] [ 3.277060] Write protecting the kernel read-only data: 28672k [ 3.279820] debug: unmapping init [mem 0xffffffff85a03000-0xffffffff85bfffff] [ 3.282747] debug: unmapping init [mem 0xffffffff86314000-0xffffffff863fffff] [ 3.309799] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.318696] systemd[1]: Detected virtualization kvm. [ 3.320053] systemd[1]: Detected architecture x86-64. [ 3.321688] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.348177] systemd[1]: No hostname configured. [ 3.350041] systemd[1]: Set hostname to . [ 3.352541] random: systemd: uninitialized urandom read (16 bytes read) [ 3.357989] systemd[1]: Initializing machine ID from random generator. [ 3.415398] random: ln: uninitialized urandom read (6 bytes read) [ 3.520469] random: systemd: uninitialized urandom read (16 bytes read) [ 3.523375] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.529264] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.534790] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.214566] device-mapper: uevent: version 1.0.3 [ 4.216690] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.022649] virtio_net virtio0 ens2: renamed from eth0 [ 5.066840] scsi host0: ata_piix [ 5.113890] scsi host1: ata_piix [ 5.116032] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.118680] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.957007] random: fast init done [ 9.413650] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.765578] random: crng init done [ 9.767132] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.173944] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.258825] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.512242] SELinux: Disabled at runtime. [ 11.573144] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.583922] systemd[1]: Detected virtualization kvm. [ 11.586470] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.102492] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.105825] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.111198] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.117269] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.122511] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.130734] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.134760] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK [[ 12.295728] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.604444] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.886792] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.911739] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.990554] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.008669] EDAC sbridge: Ver: 1.1.2 [ 14.114816] Key type dns_resolver registered [ 14.430987] NFS: Registering the id_resolver key type [ 14.434513] Key type id_resolver registered [ 14.436162] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-client login: [ 47.584197] hrtimer: interrupt took 3189784 ns [ 66.209072] libcfs: loading out-of-tree module taints kernel. [ 66.478073] Key type ._llcrypt registered [ 66.480307] Key type .llcrypt registered [ 67.294320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 67.308800] alg: No test for adler32 (adler32-zlib) [ 68.978859] Lustre: Lustre: Build Version: 2.17.57_103_ga9ee299 [ 69.598243] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 71.303648] Key type lgssc registered [ 73.180748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 184.672753] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299 [ 188.675524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.502231] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.000161] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing yml_node [ 206.132317] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 208.228117] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 210.298045] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 210.400737] Lustre: lustre-OST0000-osc-ffff941dc7dd0000: disconnect after 24s idle [ 211.758300] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 1 07:45:14 EDT 2026 [ 225.963399] Lustre: Unmounted lustre-client [ 257.457960] Key type lgssc unregistered [ 257.782281] LNet: 8400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 257.790844] LNetError: 8400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 257.814904] LNet: Removed LNI 192.168.203.45@tcp [ 258.541711] Key type .llcrypt unregistered [ 258.545579] Key type ._llcrypt unregistered [ 282.441637] Key type ._llcrypt registered [ 282.449085] Key type .llcrypt registered [ 282.643161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 282.657736] alg: No test for adler32 (adler32-zlib) [ 283.538483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 283.593870] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 283.626295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 287.542460] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 287.801888] LNet: 9279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.808360] LNetError: 9279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 287.824880] LNet: Removed LNI 192.168.203.45@tcp [ 288.538349] Key type .llcrypt unregistered [ 288.539981] Key type ._llcrypt unregistered [ 294.588762] Key type ._llcrypt registered [ 294.596590] Key type .llcrypt registered [ 295.842878] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:46:38 (1788263198) [ 296.262252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 296.272492] alg: No test for adler32 (adler32-zlib) [ 297.182322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.244720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.262354] LNet: 9687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.286791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.381795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.404939] LNet: 9694:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 304.765910] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:46:47 (1788263207) [ 305.213848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.249549] LNet: 10271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.288307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.406064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 316.625677] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:46:58 (1788263218) [ 316.959893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.013549] LNet: 10853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.055303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.194785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 326.164130] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:47:08 (1788263228) [ 326.379140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.406182] LNet: 11441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.434469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.502530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 333.385339] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:47:16 (1788263236) [ 333.543671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.572871] LNet: 12025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.590885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.688018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 334.307414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 342.495651] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:47:24 (1788263244) [ 342.637240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.689762] LNet: 12610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.727687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.888973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 343.713666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 344.638181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 353.087711] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:47:35 (1788263255) [ 353.305216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.369859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.462344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 360.191996] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:47:43 (1788263263) [ 360.522387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.553995] LNet: 13802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.564944] LNet: 13802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 360.598820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.691166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 360.740725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.202347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 361.236692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 361.583515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 361.622497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 362.011085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 362.062766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 362.603850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 362.666588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 363.216609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 363.268948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.024413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 364.088630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.857871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 364.911966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 365.713597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 365.739811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 366.238641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 366.303776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 366.853476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 366.908264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 367.281876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 367.313333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 367.554809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 367.587495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 374.421814] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:47:57 (1788263277) [ 374.578066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.630427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.766815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 374.799787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 382.491839] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:48:05 (1788263285) [ 382.603682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.645673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.783280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 382.826660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 390.270040] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:48:12 (1788263292) [ 390.520514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.591455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.708465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 390.773816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 398.537281] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:48:21 (1788263301) [ 398.655284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.699460] LNet: 16238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.704379] LNet: 16238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 398.746748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.824694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 398.854142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 406.905287] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:48:29 (1788263309) [ 407.135425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.217138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.358491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 407.390557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 415.989274] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:48:38 (1788263318) [ 416.200573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.317076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.512118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 416.646985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 425.653418] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:48:48 (1788263328) [ 425.812358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.841884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.953351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.295368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.870874] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:48:55 (1788263335) [ 433.053463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.091519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.330760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.511650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.499470] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:49:03 (1788263343) [ 440.890865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.945191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.082080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.259240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.988385] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:49:10 (1788263350) [ 447.066187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.123939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.585389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.847586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.909754] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:49:16 (1788263356) [ 453.980116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.029927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.106771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.143383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.256503] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:49:23 (1788263363) [ 460.530185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.614495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.721130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.794373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.741568] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:49:30 (1788263370) [ 467.912971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.934869] LNet: 21686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.946686] LNet: 21686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 467.974179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.071408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 474.441729] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:49:37 (1788263377) [ 474.718278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.776392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.835835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 480.318883] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:49:43 (1788263383) [ 480.469243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.541748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.625250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 481.004401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 486.698578] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:49:49 (1788263389) [ 486.824760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.861359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.949564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 486.974437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 492.518639] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:49:55 (1788263395) [ 492.683458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.732715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.807960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 492.829878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 498.474860] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:50:01 (1788263401) [ 498.643024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.698649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.760226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 498.794021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 504.564120] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:50:07 (1788263407) [ 504.796664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.823763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.903279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 504.947319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 510.539578] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:50:13 (1788263413) [ 510.767799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.812106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.859438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 510.908738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 510.927860] LNetError: 25802:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 510.985496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 511.031097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 516.021479] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:50:19 (1788263419) [ 516.090948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.133491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.224710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Vxr9m4EsJF [ 516.297816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Vxr9m4EsJF [ 516.364211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Vxr9m4EsJF [ 520.974285] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:50:24 (1788263424) [ 521.185104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.229892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.289978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 521.350462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 521.411515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 526.317351] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:50:29 (1788263429) [ 526.448276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.522754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.604052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.641406] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 527.907863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 528.867207] Lustre: Lustre: Build Version: 2.17.57_103_ga9ee299 [ 530.735173] Key type lgssc registered [ 531.770667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 535.794398] LNetError: 27562:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.45@tcp99 has no available nets [ 538.351507] Key type lgssc unregistered [ 538.704582] LNetError: 28668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.725773] LNet: Removed LNI 192.168.203.45@tcp [ 539.300391] Key type .llcrypt unregistered [ 539.303788] Key type ._llcrypt unregistered [ 545.038400] Key type ._llcrypt registered [ 545.049770] Key type .llcrypt registered [ 546.137439] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:50:49 (1788263449) [ 546.577634] Key type .llcrypt unregistered [ 546.579604] Key type ._llcrypt unregistered [ 551.435781] Key type ._llcrypt registered [ 551.441278] Key type .llcrypt registered [ 551.557972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 551.570620] alg: No test for adler32 (adler32-zlib) [ 552.499526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.546986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 552.572833] LNet: 29578:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 552.580314] LNet: 29578:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 552.592748] LNet: 29578:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 552.605446] LNet: Message forwarding will be enabled in 2s [ 555.680513] LNet: Message forwarding enabled [ 556.145975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.165204] LNet: Message forwarding will stop in 2s to 6s [ 559.780658] LNet: Message forwarding disabled [ 560.921237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.953534] LNet: 29677:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 560.963329] LNet: 29677:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 560.972566] LNet: 29677:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 560.984807] LNet: Message forwarding will be enabled in 2s [ 561.202288] LNet: 29690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.108572] Key type .llcrypt unregistered [ 562.112162] Key type ._llcrypt unregistered [ 573.164724] Key type ._llcrypt registered [ 573.171778] Key type .llcrypt registered [ 574.519484] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:51:17 (1788263477) [ 574.983342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 575.011681] alg: No test for adler32 (adler32-zlib) [ 575.924769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.999362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 581.549344] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:51:24 (1788263484) [ 581.756801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.780890] LNet: 31090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.799095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.856833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 581.897324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 581.930538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 581.982343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 582.025580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 582.083632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 582.114754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.153376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 582.188822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 582.225359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 582.273630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 582.309196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 582.345814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 582.373638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 582.407828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 582.443459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 582.485123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 582.527261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.564252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 582.589437] LNetError: 31133:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 588.525172] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:51:31 (1788263491) [ 588.732536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.767257] LNet: 31704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.782197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.841569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.225833] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:51:37 (1788263497) [ 595.627534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg345-server [ 597.227635] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:51:40 (1788263500) [ 597.411042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.466580] LNet: 32635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.487779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.555808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.582315] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 598.287682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[46-50]@tcp [ 598.302260] LNet: 32768:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 598.339962] LNetError: 32769:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.49@tcp has gone from up to down [ 598.356332] LNetError: 32639:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.48@tcp is being used as a gateway but routing feature is not turned on [ 603.987345] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:51:46 (1788263506) [ 605.446908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg345-server [ 606.639476] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:51:49 (1788263509) [ 606.847917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.872390] LNet: 33704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.879212] LNetError: 33704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.914217] LNet: Removed LNI 192.168.203.45@tcp [ 606.953925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.054381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.079620] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 607.831757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 607.865526] LNet: 33821:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 607.886030] LNet: 33821:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 607.910217] LNetError: 33708:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.50@tcp is being used as a gateway but routing feature is not turned on [ 607.913193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 607.922824] LNetError: 33708:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 607.944931] LNetError: 33708:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.50@tcp has gone from up to down [ 607.951679] LNetError: 33708:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 613.695686] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:51:56 (1788263516) [ 613.855789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.892703] LNet: 34413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.900727] LNetError: 34413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.956172] LNet: Removed LNI 192.168.203.45@tcp [ 614.988230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.102253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 615.169935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.218767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.332386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.405784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.453995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.528447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 615.604175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.653537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.753504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.852549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.890851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.936960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 615.991361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.022672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.063465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.115134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.144563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.193386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 616.258414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.316058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.410227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.514726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.563782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.625437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.706782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.739181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.800555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.873102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.914277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.971996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 617.049585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.077582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.168643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.248634] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:52:05 (1788263525) [ 623.793643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg345-server [ 625.051659] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:52:08 (1788263528) [ 626.456924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg345-server [ 627.973638] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:52:10 (1788263530) [ 628.080290] LNet: 35806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.090353] LNet: 35806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 628.568479] Key type .llcrypt unregistered [ 628.569989] Key type ._llcrypt unregistered [ 633.488733] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 633.583263] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 633.600133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 633.609133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 636.035181] Key type ._llcrypt registered [ 636.036972] Key type .llcrypt registered [ 636.154033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 636.169486] alg: No test for adler32 (adler32-zlib) [ 637.045392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.092195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 637.121328] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 637.724386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 637.743987] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 639.644279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 639.678853] LNet: Removed LNI 10.1.2.3@tcp [ 639.824407] LNet: 36525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.832465] LNetError: 36525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.862362] LNet: Removed LNI 192.168.203.45@tcp [ 640.542568] Key type .llcrypt unregistered [ 640.546973] Key type ._llcrypt unregistered [ 652.936883] Key type ._llcrypt registered [ 652.942127] Key type .llcrypt registered [ 654.748526] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:52:37 (1788263557) [ 655.296499] Key type .llcrypt unregistered [ 655.298269] Key type ._llcrypt unregistered [ 662.242323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 662.355970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 662.358340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 662.360717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.966110] Key type ._llcrypt registered [ 663.971406] Key type .llcrypt registered [ 664.313299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.342762] alg: No test for adler32 (adler32-zlib) [ 665.387689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.523072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.566675] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 666.722524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 666.754921] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 670.262360] LNet: 38007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.275871] LNetError: 38007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 670.280795] LNet: Removed LNI 10.1.2.3@tcp [ 670.295028] LNet: Removed LNI 192.168.203.45@tcp [ 670.955556] Key type .llcrypt unregistered [ 670.964413] Key type ._llcrypt unregistered [ 683.852276] Key type ._llcrypt registered [ 683.853963] Key type .llcrypt registered [ 685.086411] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:53:07 (1788263587) [ 685.517296] Key type .llcrypt unregistered [ 685.519471] Key type ._llcrypt unregistered [ 690.747939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 690.850938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.858980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.863926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 693.452527] Key type ._llcrypt registered [ 693.454371] Key type .llcrypt registered [ 693.619257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.632579] alg: No test for adler32 (adler32-zlib) [ 694.526000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.572694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 694.592461] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 696.098887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 696.126663] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.097860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 698.122585] LNet: Removed LNI 10.1.2.3@tcp [ 698.140963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 698.173862] LNet: Removed LNI 10.1.2.31@tcp [ 698.205477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.248152] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.858337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 698.879250] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 699.834515] LNet: 39925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.847628] LNetError: 39925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 699.857897] LNet: Removed LNI 10.1.2.31@tcp [ 699.877889] LNet: Removed LNI 10.1.2.3@tcp [ 700.548087] Key type .llcrypt unregistered [ 700.550556] Key type ._llcrypt unregistered [ 710.761871] Key type ._llcrypt registered [ 710.764319] Key type .llcrypt registered [ 712.112974] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:53:34 (1788263614) [ 712.591484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.618570] alg: No test for adler32 (adler32-zlib) [ 713.564995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.605218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.633564] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 714.616769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.644916] LNet: 40876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.651088] LNetError: 40876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.668165] LNet: Removed LNI 192.168.203.45@tcp [ 714.677491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.709370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.45@tcp [ 714.727673] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 715.755399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.45@tcp [ 715.795886] LNet: Removed LNI 192.168.203.45@tcp [ 715.874380] LNet: 41011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.561546] Key type .llcrypt unregistered [ 716.564070] Key type ._llcrypt unregistered [ 728.646888] Key type ._llcrypt registered [ 728.649311] Key type .llcrypt registered [ 730.207121] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:53:52 (1788263632) [ 732.067552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg345-server [ 734.161681] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:53:56 (1788263636) [ 734.934393] Key type .llcrypt unregistered [ 734.938284] Key type ._llcrypt unregistered [ 742.178517] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 742.305219] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 742.324551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 742.339591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 744.980178] Key type ._llcrypt registered [ 744.984276] Key type .llcrypt registered [ 745.162391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.178100] alg: No test for adler32 (adler32-zlib) [ 746.084482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.167692] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 746.724267] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 749.274339] LNet: 42841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.285461] LNetError: 42841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.293403] LNet: Removed LNI 192.168.203.45@tcp1 [ 749.302263] LNet: Removed LNI 10.1.2.3@tcp2 [ 750.030968] Key type .llcrypt unregistered [ 750.037671] Key type ._llcrypt unregistered [ 761.924837] Key type ._llcrypt registered [ 761.929779] Key type .llcrypt registered [ 763.760391] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:54:25 (1788263665) [ 764.247458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.257608] alg: No test for adler32 (adler32-zlib) [ 765.198763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.275434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.291265] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 766.452926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.484742] LNet: 43808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.497756] LNetError: 43808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.517751] LNet: Removed LNI 192.168.203.45@tcp [ 766.536555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.644473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 766.704285] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 773.708528] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:54:36 (1788263676) [ 773.827984] LNet: 44392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.834239] LNetError: 44392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.862476] LNet: Removed LNI 192.168.203.45@tcp [ 774.411286] Key type .llcrypt unregistered [ 774.413093] Key type ._llcrypt unregistered [ 781.458042] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.767497] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.774304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.784313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.356594] Key type ._llcrypt registered [ 784.358957] Key type .llcrypt registered [ 784.607573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.628641] alg: No test for adler32 (adler32-zlib) [ 785.584937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.629396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.672118] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 786.737481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 786.762532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.642339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.678296] LNet: 45137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.691034] LNetError: 45137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 789.700316] LNet: Removed LNI 10.1.2.3@tcp [ 789.720635] LNet: Removed LNI 192.168.203.45@tcp [ 789.783648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.889327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 789.953168] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 789.961590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.634991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.665700] LNet: 45165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.681860] LNetError: 45165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 790.689688] LNetError: 45165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 790.698737] LNet: Removed LNI 10.1.2.3@tcp [ 790.723947] LNet: Removed LNI 192.168.203.45@tcp [ 790.758530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.834180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 790.874120] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 790.881522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.316672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 791.341772] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 792.348274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.371877] LNet: 45327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.376645] LNetError: 45327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.381508] LNetError: 45327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.388315] LNet: Removed LNI 10.1.2.3@tcp [ 792.390635] LNet: Removed LNI 192.168.203.45@tcp [ 792.406403] LNet: Removed LNI 192.168.203.45@tcp2 [ 792.423914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.501669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 792.526937] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 792.530745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.540266] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 800.743833] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:55:03 (1788263703) [ 800.979265] LNet: 45922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.998698] LNetError: 45922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.015784] LNetError: 45922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 801.027075] LNet: Removed LNI 10.1.2.3@tcp [ 801.039562] LNet: Removed LNI 192.168.203.45@tcp [ 801.086288] LNet: Removed LNI 192.168.203.45@tcp2 [ 801.954479] Key type .llcrypt unregistered [ 801.961836] Key type ._llcrypt unregistered [ 807.793521] Key type ._llcrypt registered [ 807.795831] Key type .llcrypt registered [ 807.934438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.950067] alg: No test for adler32 (adler32-zlib) [ 808.902181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.982870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.000113] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 810.072082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.111843] LNet: 46317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.120323] LNetError: 46317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.138469] LNet: Removed LNI 192.168.203.45@tcp [ 810.158109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.263255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 810.296432] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 810.648559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.666383] LNet: 46347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.680123] LNetError: 46347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.712131] LNet: Removed LNI 192.168.203.45@tcp [ 810.740536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.820299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 810.858427] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 811.320935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.338992] LNet: Removed LNI 192.168.203.45@tcp [ 811.363251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.462917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 811.509812] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 812.185225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.362289] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.365613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.368523] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.786457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.809986] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.815156] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 814.825359] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.834203] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 814.854708] LNet: Removed LNI 192.168.203.45@tcp [ 814.913270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.991461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 815.018101] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 824.470248] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:55:26 (1788263726) [ 824.922936] LNet: 47053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.950189] LNetError: 47053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.968447] LNet: Removed LNI 192.168.203.45@tcp [ 825.984685] Key type .llcrypt unregistered [ 825.998347] Key type ._llcrypt unregistered [ 832.724298] Key type ._llcrypt registered [ 832.728940] Key type .llcrypt registered [ 833.002668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.024470] alg: No test for adler32 (adler32-zlib) [ 833.979897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.044691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.065821] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 835.182829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.207896] LNet: 47449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.218793] LNetError: 47449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.236692] LNet: Removed LNI 192.168.203.45@tcp [ 835.268700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.369682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 835.413717] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 842.458727] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:55:45 (1788263745) [ 842.648386] LNet: 48035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.655211] LNetError: 48035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.677700] LNet: Removed LNI 192.168.203.45@tcp [ 843.242580] Key type .llcrypt unregistered [ 843.244086] Key type ._llcrypt unregistered [ 849.505817] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.754407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.764737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.767385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.367418] Key type ._llcrypt registered [ 852.372730] Key type .llcrypt registered [ 852.618789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.631921] alg: No test for adler32 (adler32-zlib) [ 853.574474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.610791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.627355] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 855.130462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.152860] LNet: 48484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.173615] LNetError: 48484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.202081] LNet: Removed LNI 192.168.203.45@tcp [ 855.228536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.317625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 855.353692] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 861.689763] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:56:04 (1788263764) [ 861.866167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.887220] LNet: 49082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.893218] LNetError: 49082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.910572] LNet: Removed LNI 192.168.203.45@tcp [ 861.943928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.024875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 867.663853] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:56:10 (1788263770) [ 867.820867] LNet: 49661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.532477] Key type .llcrypt unregistered [ 868.537265] Key type ._llcrypt unregistered [ 873.944153] Key type ._llcrypt registered [ 873.946975] Key type .llcrypt registered [ 874.143669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.164135] alg: No test for adler32 (adler32-zlib) [ 875.106034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.167410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.220199] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 875.827429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.846684] LNet: 50058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.857872] LNetError: 50058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.893261] LNet: Removed LNI 192.168.203.45@tcp [ 875.921963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.994201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 876.016926] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 882.549648] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:56:25 (1788263785) [ 882.661777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.692570] LNet: 50645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.710705] LNetError: 50645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.733839] LNet: Removed LNI 192.168.203.45@tcp [ 882.755719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.838692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 890.639880] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:56:32 (1788263792) [ 892.303939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 894.749034] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:56:36 (1788263796) [ 896.871350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 898.687981] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:56:41 (1788263801) [ 901.067468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 903.101643] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:56:45 (1788263805) [ 905.275870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 907.118637] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:56:49 (1788263809) [ 908.547386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 910.415656] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:56:53 (1788263813) [ 910.482617] LNet: 52697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.265559] Key type .llcrypt unregistered [ 911.267941] Key type ._llcrypt unregistered [ 917.313530] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.520577] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.523330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.527895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.927869] Key type ._llcrypt registered [ 919.929976] Key type .llcrypt registered [ 920.144432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.156830] alg: No test for adler32 (adler32-zlib) [ 921.021130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.175958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.229650] LNet: Added LNI 192.168.203.45@tcp [8/512/0/180] [ 921.243510] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 921.254450] LNet: 53020:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.391459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.409887] LNet: 53039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.417902] LNetError: 53039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.430295] LNet: Removed LNI 10.1.2.3@tcp [ 921.433326] LNetError: 53039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.451574] LNet: Removed LNI 192.168.203.45@tcp [ 921.480027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.644566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.697387] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 921.830772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.869510] LNet: 53063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.873947] LNetError: 53063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.892220] LNet: Removed LNI 192.168.203.45@tcp [ 921.917907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.986146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.020264] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 922.033708] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.039249] LNet: 53071:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 922.132216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.153395] LNet: 53089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.158700] LNetError: 53089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.165589] LNet: Removed LNI 10.1.2.3@tcp [ 922.168514] LNetError: 53089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.195703] LNet: Removed LNI 192.168.203.45@tcp [ 922.237344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.360504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.386099] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 922.393137] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.395973] LNet: 53097:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 922.558183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.579846] LNet: 53123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.586659] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.592965] LNet: Removed LNI 10.1.2.3@tcp [ 922.596773] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.611818] LNet: Removed LNI 192.168.203.45@tcp [ 922.649210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.723668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.757332] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 922.760933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.844909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.862434] LNet: 53151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.867697] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.873715] LNet: Removed LNI 10.1.2.3@tcp [ 922.876082] LNetError: 53151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.889504] LNet: Removed LNI 192.168.203.45@tcp [ 922.903263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.956895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.978299] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 922.983546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.988287] LNet: 53159:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 923.063453] LNet: Removed LNI 10.1.2.3@tcp [ 923.073037] LNet: Removed LNI 192.168.203.45@tcp [ 923.601036] Key type .llcrypt unregistered [ 923.603155] Key type ._llcrypt unregistered [ 935.116507] Key type ._llcrypt registered [ 935.117984] Key type .llcrypt registered [ 936.861979] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:57:19 (1788263839) [ 938.632452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 940.188562] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:57:23 (1788263843) [ 940.739283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.758357] alg: No test for adler32 (adler32-zlib) [ 941.759781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.829146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.882158] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 943.123599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.45 [ 943.143110] LNet: 54421:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 943.151765] LNet: Removed LNI 192.168.203.45@tcp [ 952.350365] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:57:34 (1788263854) [ 952.534432] LNet: 54984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.557153] LNetError: 54984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.588667] LNet: Removed LNI 192.168.203.45@tcp [ 953.577535] Key type .llcrypt unregistered [ 953.588379] Key type ._llcrypt unregistered [ 960.809977] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 961.061920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.742794] Key type ._llcrypt registered [ 963.746540] Key type .llcrypt registered [ 964.018363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.048688] alg: No test for adler32 (adler32-zlib) [ 965.276613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 972.294180] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:57:54 (1788263874) [ 972.442470] LNet: 55893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.454938] LNetError: 55893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 972.487969] LNet: Removed LNI 10.1.2.3@tcp [ 973.321422] Key type .llcrypt unregistered [ 973.327602] Key type ._llcrypt unregistered [ 980.179347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.441559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 980.445614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 980.451842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.077479] Key type ._llcrypt registered [ 983.080515] Key type .llcrypt registered [ 983.231431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.249667] alg: No test for adler32 (adler32-zlib) [ 984.211571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.038486] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:58:12 (1788263892) [ 991.817641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 993.377727] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:58:16 (1788263896) [ 993.545686] LNet: 57084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.557745] LNetError: 57084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 993.582493] LNet: Removed LNI 10.1.2.3@tcp [ 994.180807] Key type .llcrypt unregistered [ 994.183734] Key type ._llcrypt unregistered [ 999.449992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.639794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.643938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.650870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.335294] Key type ._llcrypt registered [ 1002.339875] Key type .llcrypt registered [ 1002.534947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.552380] alg: No test for adler32 (adler32-zlib) [ 1003.407481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.486996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1013.378630] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:58:36 (1788263916) [ 1013.893831] LNet: 58312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.899268] LNetError: 58312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.918098] LNet: Removed LNI 10.1.2.3@tcp [ 1014.637580] Key type .llcrypt unregistered [ 1014.639331] Key type ._llcrypt unregistered [ 1023.357937] Key type ._llcrypt registered [ 1023.359742] Key type .llcrypt registered [ 1023.633783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.667373] alg: No test for adler32 (adler32-zlib) [ 1024.574846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1024.604274] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1033.444111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1034.158035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1034.181556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1034.228512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1034.255847] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1034.463210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.496096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.515811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.545788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.604270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.646338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.673506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.715200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.733983] LNet: 58747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.757454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.795984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.839670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.853104] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1034.876688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.912635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.979811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.039201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.775709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1035.834188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1035.884737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1035.921133] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1036.190781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.223562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1036.283261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.365372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.436502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.500526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.557509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.600979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.635502] LNet: 58856:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.653209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.711955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.757534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.793376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.843580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.882222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.923311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1037.524901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1037.575520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1037.642741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1037.674284] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1037.684099] LNet: Skipped 1 previous similar message [ 1037.958392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.984849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1038.013985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.046244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.086607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.127029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.153829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1038.201180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1038.239717] LNet: 58966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1038.268650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1038.339400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1038.370660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1038.411577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1038.451841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1038.502516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.547605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1039.129120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1039.187587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1039.275727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1039.567942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.624530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1039.653529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.692125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.716122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.742236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.776578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.812449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.860879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.962603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.019791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.050717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.102391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.146471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.196151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.717665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1040.755394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1040.789210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1040.804374] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1040.809678] LNet: Skipped 1 previous similar message [ 1041.126910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.184960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.235023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.278556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.322283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.383127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.427932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.465162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.526237] LNet: 59184:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.534736] LNet: 59184:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1041.591956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.645800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.711932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.771814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.833174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.887613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.937367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1042.558774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1042.594928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1042.624953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1042.869353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.910287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1042.974235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.022469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.084880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.127082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.172987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.222937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.261953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.301691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.337960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.381935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.427669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.472437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.504611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.818819] LNet: 59334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.832874] LNetError: 59334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.904760] LNet: Removed LNI 192.168.203.45@tcp [ 1045.528542] Key type .llcrypt unregistered [ 1045.531824] Key type ._llcrypt unregistered [ 1063.699409] Key type ._llcrypt registered [ 1063.701397] Key type .llcrypt registered [ 1065.260575] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:59:27 (1788263967) [ 1066.083441] Key type .llcrypt unregistered [ 1066.094408] Key type ._llcrypt unregistered [ 1073.051603] Key type ._llcrypt registered [ 1073.056603] Key type .llcrypt registered [ 1073.305987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.325920] alg: No test for adler32 (adler32-zlib) [ 1074.337262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1074.362389] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1082.624257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1094.835447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.894313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1094.933987] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1096.669955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1096.740798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 1096.794181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1096.836270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 1096.890656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1096.940439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 1097.003344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1097.059651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 1097.103183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1097.127921] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1097.134638] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1097.435117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.463634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.521599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.570812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.596394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.627559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.655034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.686826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.703664] LNet: 60798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.724478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.748276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.778883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.808385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.837792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.865673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.893705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.531886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1098.567790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 1098.597585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1098.629569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 1098.662856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1098.697980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 1098.742856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1098.810421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 1098.864036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1098.887630] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1098.895149] LNet: Skipped 2 previous similar messages [ 1099.164434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.195210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.218932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.252979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.284181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.315171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.343776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.388158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.406892] LNet: 60920:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.443279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.488127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.521231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.561151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.587393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.616191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.641530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.292225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1100.331793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 1100.406095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1100.485124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 1100.555330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1100.611875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 1100.684809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1100.727478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 1100.772842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1100.794292] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1100.801247] LNet: Skipped 2 previous similar messages [ 1101.087735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.121393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.172647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.213078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.247741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.283467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.322489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.360265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.379950] LNet: 61041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.399676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.432572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.485423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.512366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.560585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.599632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.641593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.290639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1102.329305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 1102.364640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1102.408948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 1102.462526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1102.514626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 1102.554792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1102.602413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 1102.657693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1102.999550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.054940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.103983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.164990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.213429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.255562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.299430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.329242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.391585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.435608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.486129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.535633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.586674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.636169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.678223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.250240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1104.275393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 1104.310511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1104.342571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 1104.377853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1104.413859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 1104.454661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1104.505753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 1104.553734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1104.578557] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1104.585993] LNet: Skipped 5 previous similar messages [ 1104.870613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.907526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.955625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.002198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.054812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.121587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.184847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.247869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.282778] LNet: 61283:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.299477] LNet: 61283:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1105.335460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.388149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.455777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.501805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.565279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.645904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.734692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.369862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1106.407775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 1106.447977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1106.495934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 1106.531960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1106.559817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 1106.589133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1106.623535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 1106.660570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1106.854413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.911781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.940614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.980751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.015607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.068518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.130809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.161935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.187438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.222249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.281257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.319374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.363911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.416808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.872982] LNet: 61481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.880636] LNetError: 61481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.913557] LNet: Removed LNI 192.168.203.45@tcp [ 1114.915694] LNetError: 61481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.948643] LNet: Removed LNI 192.168.203.45@tcp1 [ 1115.413778] Key type .llcrypt unregistered [ 1115.420392] Key type ._llcrypt unregistered [ 1133.497768] Key type ._llcrypt registered [ 1133.499931] Key type .llcrypt registered [ 1134.867429] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:00:37 (1788264037) [ 1135.785456] Key type .llcrypt unregistered [ 1135.790054] Key type ._llcrypt unregistered [ 1142.529647] Key type ._llcrypt registered [ 1142.533064] Key type .llcrypt registered [ 1142.720546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.735174] alg: No test for adler32 (adler32-zlib) [ 1143.640765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.672394] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1151.663796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1152.486719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1152.540262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1152.590837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1152.641787] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1152.926462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.985785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.080619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.144504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.214037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.302243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.363507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.435449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.460559] LNet: 62724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.494652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.534030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.588914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.647918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.712791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.752934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.801366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1154.310739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1154.346906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1154.377985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1154.393759] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1154.632928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.660500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.688211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.737937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.810887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.872902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.927683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.019595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.073047] LNet: 62834:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.103212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.163856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.233740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.290804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.331796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.371255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.389717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.247266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1156.291844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1156.356809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1156.385740] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1156.754462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.802742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.847803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.883613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.935793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.987986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.050769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.123643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.143823] LNet: 62943:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.163563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.208375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.287357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.332782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.373187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.420415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.500420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.053259] LNet: 62997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.063961] LNetError: 62997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.098501] LNet: Removed LNI 192.168.203.45@tcp [ 1159.010982] Key type .llcrypt unregistered [ 1159.016875] Key type ._llcrypt unregistered [ 1179.271341] Key type ._llcrypt registered [ 1179.274457] Key type .llcrypt registered [ 1180.882545] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:01:23 (1788264083) [ 1182.040681] Key type .llcrypt unregistered [ 1182.043886] Key type ._llcrypt unregistered [ 1189.269740] Key type ._llcrypt registered [ 1189.272375] Key type .llcrypt registered [ 1189.421180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1189.439913] alg: No test for adler32 (adler32-zlib) [ 1190.475676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1190.501803] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1199.153099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1208.757536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.849487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1208.909687] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1210.677683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1210.714951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1210.767995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1210.817309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1210.848893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1210.878888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1210.918495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1210.948441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1210.979243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1210.997478] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1211.002904] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1211.249088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.285606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.328236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.361738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.419506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.453341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.488604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.518852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.539898] LNet: 64459:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.567841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.605945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.637697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.665603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.701452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.752234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.790394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1211.815630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.429665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1212.469985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1212.515844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1212.561941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1212.609688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1212.642877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1212.701687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1212.760787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1212.802892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1212.842783] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1212.850495] LNet: Skipped 2 previous similar messages [ 1213.167403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.189547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1213.213364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.238315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.282888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.341573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.393611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.449715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.481291] LNet: 64582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.511191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.544516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.571688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.628499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.661369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.696459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.745754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.793918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.466761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1214.495504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1214.523741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1214.552733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1214.582877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1214.613376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1214.643428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1214.674872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1214.702603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1214.716691] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1214.721701] LNet: Skipped 1 previous similar message [ 1214.975655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.000439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.022478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.047446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.073536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.107064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.142571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.183973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.208619] LNet: 64706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.225433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.261649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.291770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.325293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.368745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.412238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.471661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1215.526869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.141831] LNet: 64784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.156972] LNetError: 64784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.171600] LNet: Removed LNI 192.168.203.45@tcp [ 1221.182129] LNet: Removed LNI 192.168.203.45@tcp1 [ 1221.663541] Key type .llcrypt unregistered [ 1221.667177] Key type ._llcrypt unregistered [ 1240.167779] Key type ._llcrypt registered [ 1240.170575] Key type .llcrypt registered [ 1241.611445] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:02:24 (1788264144) [ 1243.129651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1244.936132] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:02:27 (1788264147) [ 1245.805451] Key type .llcrypt unregistered [ 1245.807596] Key type ._llcrypt unregistered [ 1251.506228] Key type ._llcrypt registered [ 1251.509756] Key type .llcrypt registered [ 1251.728404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.750815] alg: No test for adler32 (adler32-zlib) [ 1252.722724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.754707] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1261.460396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1262.161537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1262.192653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1262.238740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1262.261249] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1262.558864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.605596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.655203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.697704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.722180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.750592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.776146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.809454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.822581] LNet: 66320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.838358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.868128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.898610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.928765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.959172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.992938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.008623] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1263.019689] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1263.365661] LNet: 66359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.377298] LNetError: 66359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.399597] LNet: Removed LNI 192.168.203.45@tcp [ 1263.908759] Key type .llcrypt unregistered [ 1263.912500] Key type ._llcrypt unregistered [ 1284.395288] Key type ._llcrypt registered [ 1284.402097] Key type .llcrypt registered [ 1284.715315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1284.739627] alg: No test for adler32 (adler32-zlib) [ 1285.795241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1285.877805] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1294.948563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1305.726337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.778851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.823499] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1307.284091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1307.349441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1307.391363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1307.461166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1307.495943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1307.556777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1307.617319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1307.694891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1307.778570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1307.829293] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1308.217104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1308.265952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1308.302715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.339867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.383746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.431223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.450285] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1308.452179] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1308.458394] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1308.471356] LNet: Skipped 1 previous similar message [ 1308.472723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.520068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.545804] LNet: 67234:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.579273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.632388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.687837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.754917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.809369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.858702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1314.040799] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1314.366579] LNet: 67308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.378903] LNetError: 67308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.390796] LNet: Removed LNI 192.168.203.45@tcp [ 1314.406310] LNet: Removed LNI 192.168.203.45@tcp1 [ 1314.805067] Key type .llcrypt unregistered [ 1314.813994] Key type ._llcrypt unregistered [ 1335.733627] Key type ._llcrypt registered [ 1335.743764] Key type .llcrypt registered [ 1337.289841] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:03:59 (1788264239) [ 1337.735922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.754172] alg: No test for adler32 (adler32-zlib) [ 1338.898272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.085205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.132303] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1340.156651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.194915] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1341.815629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1341.947889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1342.003652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1342.033655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1342.088423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1342.157948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1342.210857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1342.236638] LNetError: 68409:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1343.072225] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1343.073695] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1345.122056] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1346.281606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1349.216159] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1350.268495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1350.299919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1350.385552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1350.411513] LNet: 68516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.421493] LNetError: 68516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.429391] LNet: Removed LNI 192.168.203.45@tcp [ 1350.449599] LNet: Removed LNI 192.168.203.45@tcp1 [ 1350.467883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.541818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.565510] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1351.472368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.523260] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1352.616294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1352.683078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.778720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1352.852875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1352.889975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1352.927533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1353.005868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1353.058844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1353.095907] LNetError: 68789:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1353.567502] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1356.777901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.711332] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1359.717833] LNet: Skipped 1 previous similar message [ 1364.815659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1364.875455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1364.922345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.095804] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:04:35 (1788264275) [ 1373.182987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.224413] LNet: 69499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.237222] LNetError: 69499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.242984] LNetError: 69499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1373.264635] LNet: Removed LNI 192.168.203.45@tcp [ 1373.293370] LNet: Removed LNI 192.168.203.45@tcp1 [ 1373.358262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.438868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.474322] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1374.310506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.330037] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1375.488879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1375.618610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1375.666240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1375.722783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1375.780513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1375.870320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1375.923983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1376.106668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.479952] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.45@tcp [ 1376.480832] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1376.499280] LNet: Skipped 1 previous similar message [ 1391.508284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1391.677553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1391.738124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.789754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 500 [ 1391.909545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.775851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.868783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1398.914512] LNet: 69997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.923776] LNetError: 69997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.927026] LNetError: 69997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1398.942636] LNet: Removed LNI 192.168.203.45@tcp [ 1398.955646] LNet: Removed LNI 192.168.203.45@tcp1 [ 1398.984262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.084239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.140661] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1400.238715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.275468] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1401.477843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1401.541588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.626632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1401.681253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1401.724872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1401.764449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.810945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.881220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1402.148759] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1402.163723] LNet: Skipped 3 previous similar messages [ 1403.350478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.681837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.733755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1412.804137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.288864] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:05:22 (1788264322) [ 1420.802354] LNet: 71034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.810594] LNetError: 71034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.826424] LNetError: 71034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.839660] LNet: Removed LNI 192.168.203.45@tcp [ 1420.864068] LNet: Removed LNI 192.168.203.45@tcp1 [ 1421.466752] Key type .llcrypt unregistered [ 1421.470237] Key type ._llcrypt unregistered [ 1428.372763] Key type ._llcrypt registered [ 1428.378977] Key type .llcrypt registered [ 1428.706684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1428.751929] alg: No test for adler32 (adler32-zlib) [ 1429.756762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1429.821268] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1440.232919] Lustre: DEBUG MARKER: Initial discovery [ 1440.290085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1442.909955] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1442.985726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1443.049997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1443.094873] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1443.121254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1443.168625] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1443.171840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -l 3 [ 1443.245043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -e local_error [ 1443.277683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m PUT -l 6 [ 1444.670568] Lustre: DEBUG MARKER: Force oleg345-client.virtnet to discover 192.168.203.145@tcp (in background) [ 1444.825077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1446.137797] Lustre: DEBUG MARKER: Force oleg345-server to discover 192.168.203.45@tcp [ 1447.903272] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1448.426237] Lustre: DEBUG MARKER: Wait for 71789 [ 1449.783672] Lustre: DEBUG MARKER: Finished wait on 71789 [ 1449.905779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1449.945806] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1449.947592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1450.214676] LNet: 72242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.221843] LNetError: 72242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.244914] LNet: Removed LNI 192.168.203.45@tcp [ 1450.643133] Key type .llcrypt unregistered [ 1450.650248] Key type ._llcrypt unregistered [ 1469.340245] Key type ._llcrypt registered [ 1469.342954] Key type .llcrypt registered [ 1471.091823] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:06:13 (1788264373) [ 1471.626535] Key type .llcrypt unregistered [ 1471.631180] Key type ._llcrypt unregistered [ 1478.713045] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1478.980734] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1478.991261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1479.001798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1481.859964] Key type ._llcrypt registered [ 1481.873627] Key type .llcrypt registered [ 1482.151647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1482.179473] alg: No test for adler32 (adler32-zlib) [ 1483.222537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.338458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.380462] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1484.587640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1484.629289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1493.346405] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:06:36 (1788264396) [ 1493.528817] LNet: 74311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.534406] LNetError: 74311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1493.541339] LNet: Removed LNI 10.1.2.3@tcp [ 1493.554336] LNet: Removed LNI 192.168.203.45@tcp [ 1494.136612] Key type .llcrypt unregistered [ 1494.139398] Key type ._llcrypt unregistered [ 1501.015445] Key type ._llcrypt registered [ 1501.018449] Key type .llcrypt registered [ 1501.253154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1501.277442] alg: No test for adler32 (adler32-zlib) [ 1502.238142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.281829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.302856] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1503.001917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1503.040731] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1503.954172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.45@tcp [ 1505.056454] LNetError: Unexpected error -22 connecting to 192.168.203.45@tcp at host 192.168.203.45:988 [ 1511.276376] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:06:54 (1788264414) [ 1511.435403] LNet: 75435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.439550] LNetError: 75435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1511.446312] LNet: Removed LNI 10.1.2.3@tcp [ 1511.480263] LNet: Removed LNI 192.168.203.45@tcp [ 1512.401826] Key type .llcrypt unregistered [ 1512.407117] Key type ._llcrypt unregistered [ 1519.825607] Key type ._llcrypt registered [ 1519.827970] Key type .llcrypt registered [ 1520.118078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.155988] alg: No test for adler32 (adler32-zlib) [ 1521.085597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.155179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1521.189049] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1522.080557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1522.123578] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1523.349776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.45@tcp1 --nid 192.168.203.45@tcp2 [ 1528.008976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp2 [ 1528.051757] LNetError: 76273:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.45@tcp1 to 192.168.203.45@tcp2 - Route table may be misconfigured [ 1528.104336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp2 192.168.203.45@tcp1 [ 1534.896633] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:07:17 (1788264437) [ 1535.012441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.071826] LNet: 76840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.086480] LNetError: 76840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.102240] LNet: Removed LNI 192.168.203.45@tcp1 [ 1536.171598] LNet: Removed LNI 192.168.203.45@tcp2 [ 1536.210650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.332729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.376629] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1537.417060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1537.464099] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1538.457710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1538.530685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.45@tcp [ 1538.588424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp1 [ 1538.649667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp -e network_timeout [ 1538.696737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -e network_timeout [ 1538.746393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -e network_timeout [ 1538.784368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -e network_timeout [ 1538.828203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1538.859467] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1538.894388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.029372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.171362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1539.362597] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1546.328530] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:07:28 (1788264448) [ 1546.474578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.524155] LNet: 77702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.538438] LNetError: 77702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.550089] LNetError: 77702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1546.557803] LNet: Removed LNI 192.168.203.45@tcp [ 1547.623964] LNet: Removed LNI 192.168.203.45@tcp1 [ 1547.657607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.794364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1548.141439] LNet: 77746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.628803] Key type .llcrypt unregistered [ 1548.631703] Key type ._llcrypt unregistered [ 1560.849533] Key type ._llcrypt registered [ 1560.854485] Key type .llcrypt registered [ 1562.360918] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:07:44 (1788264464) [ 1562.738651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.755480] alg: No test for adler32 (adler32-zlib) [ 1563.656847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.165192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1567.217628] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:07:49 (1788264469) [ 1567.345322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.376208] LNet: 78835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.410852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.517980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.547283] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1568.369943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1568.419419] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1569.438460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1569.466707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp1 [ 1569.509568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp1 [ 1575.654638] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:07:58 (1788264478) [ 1577.654317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg345-server [ 1579.508158] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:08:02 (1788264482) [ 1581.818985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg345-server [ 1583.445320] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:08:06 (1788264486) [ 1585.400731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg345-server [ 1587.065763] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:08:09 (1788264489) [ 1588.729785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg345-server [ 1590.489191] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:08:13 (1788264493) [ 1592.083048] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg345-server [ 1593.696922] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:08:16 (1788264496) [ 1595.645851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg345-server [ 1597.668475] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:08:20 (1788264500) [ 1599.868762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg345-server [ 1602.034776] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:08:24 (1788264504) [ 1603.988499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg345-server [ 1605.848716] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:08:28 (1788264508) [ 1608.019491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg345-server [ 1609.587752] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:08:32 (1788264512) [ 1609.955894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.002429] LNet: 82719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.008745] LNetError: 82719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.024318] LNet: Removed LNI 192.168.203.45@tcp [ 1611.046139] LNet: Removed LNI 192.168.203.45@tcp1 [ 1611.127530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.222139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.261530] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1612.120619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1612.203211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1612.420737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.456745] LNet: 82866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.465102] LNetError: 82866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.473565] LNetError: 82866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.500986] LNet: Removed LNI 192.168.203.45@tcp [ 1612.525100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.595491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1612.628996] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1613.335566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1613.530165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.557188] LNetError: 83011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.629063] LNet: Removed LNI 192.168.203.45@tcp [ 1614.710876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.792416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.829366] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1615.534918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1615.646305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1615.818977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.859319] LNet: 83159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.862234] LNet: 83159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1615.875051] LNetError: 83159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.906260] LNet: Removed LNI 192.168.203.45@tcp [ 1615.928604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.997874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1616.032435] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1616.864535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1617.160358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.289373] LNet: Removed LNI 192.168.203.45@tcp [ 1618.353792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.461097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.510947] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1619.762657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1619.869220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1620.111364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.136185] LNet: 83451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.142120] LNet: 83451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1620.152202] LNetError: 83451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.160352] LNetError: 83451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1621.251386] LNet: Removed LNI 192.168.203.45@tcp [ 1621.300349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.397494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1621.452618] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1622.248176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1622.529378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.658649] LNet: Removed LNI 192.168.203.45@tcp [ 1623.701834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.779475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.811870] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1624.712734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1624.786814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1625.030856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.089176] LNet: Removed LNI 192.168.203.45@tcp [ 1626.120887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.176862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1626.210839] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1627.137720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1628.775355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.803291] LNet: 83898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.813905] LNet: 83898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1628.822489] LNetError: 83898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.832653] LNetError: 83898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1629.862753] LNet: Removed LNI 192.168.203.45@tcp [ 1629.903683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.004131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.054714] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1631.031245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1631.168843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1631.525694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.620783] LNet: Removed LNI 192.168.203.45@tcp [ 1632.685892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.788745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1632.850777] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1633.822396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1634.168116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.238775] LNet: Removed LNI 192.168.203.45@tcp [ 1635.301367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.402285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.464616] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1636.597825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1636.747249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1637.096874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.184343] LNet: Removed LNI 192.168.203.45@tcp [ 1638.244849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.388406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1638.430053] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1639.542953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1639.924788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.997956] LNet: Removed LNI 192.168.203.45@tcp [ 1641.069543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.187618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.222958] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1642.422326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1642.522568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.618098] LNet: Removed LNI 192.168.203.45@tcp [ 1642.677071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.762121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.781207] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1643.931451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1644.102438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.174353] LNet: Removed LNI 192.168.203.45@tcp [ 1644.220706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.290372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1644.413771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.499816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.607740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1644.787280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.825995] LNet: 84780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.835889] LNet: 84780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1644.864794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.943120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1645.050957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.094852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.156957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1645.180622] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1646.131738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.154593] LNetError: 84924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.160766] LNetError: 84924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1646.175635] LNet: Removed LNI 192.168.203.45@tcp [ 1646.198427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.279339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1653.616420] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:09:16 (1788264556) [ 1653.774427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.838225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.925980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1653.952380] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1655.673043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.721905] LNet: Removed LNI 192.168.203.45@tcp231 [ 1655.752687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.858755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1655.892383] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1655.901920] LNet: 85652:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1656.604089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1656.666050] LNet: Removed LNI 192.168.203.45@tcp231 [ 1656.694605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1656.743362] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1658.272197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.310724] LNet: Removed LNI 192.168.203.45@tcp231 [ 1658.344670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.423713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1658.452025] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1658.459671] LNet: 85805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.028828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1659.081732] LNet: Removed LNI 192.168.203.45@tcp231 [ 1659.119375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1659.145712] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1660.492588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.537904] LNet: Removed LNI 192.168.203.45@tcp231 [ 1660.558672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.667794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1660.710672] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1660.723395] LNet: 85956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.285878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1661.321948] LNet: Removed LNI 192.168.203.45@tcp231 [ 1661.349893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1661.396538] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1662.827310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.884839] LNet: Removed LNI 192.168.203.45@tcp231 [ 1662.912310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.016212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1663.037406] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1663.046424] LNet: 86107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.464068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1663.498760] LNet: Removed LNI 192.168.203.45@tcp231 [ 1663.511747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1663.539399] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1672.196897] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:09:34 (1788264574) [ 1672.328730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.420549] LNet: Removed LNI 192.168.203.45@tcp231 [ 1672.483059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.604296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1672.624886] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1673.537571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1673.600987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1673.645986] LNet: Removed LNI 192.168.203.45@tcp232 [ 1673.669138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1673.706038] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1674.476196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1674.530616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1674.568315] LNet: Removed LNI 192.168.203.45@tcp232 [ 1674.595690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1674.635123] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1675.478482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1675.520204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1675.562617] LNet: Removed LNI 192.168.203.45@tcp232 [ 1675.575521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1675.604520] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1676.467502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1676.513601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1676.551389] LNet: Removed LNI 192.168.203.45@tcp232 [ 1676.574045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1683.486826] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:09:46 (1788264586) [ 1684.047973] LNet: 87946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.053420] LNet: 87946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1684.805246] Key type .llcrypt unregistered [ 1684.808274] Key type ._llcrypt unregistered [ 1691.236675] Key type ._llcrypt registered [ 1691.244426] Key type .llcrypt registered [ 1691.463813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1691.487291] alg: No test for adler32 (adler32-zlib) [ 1692.504393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1692.553903] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1700.445569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1710.151868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.218886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.260361] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1711.810563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1711.853798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1711.937883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1711.990433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1712.062938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1712.121919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1712.175902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1712.210672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1712.244649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1712.267560] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1712.273474] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1712.567116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1712.601991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1712.658456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1712.708422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1712.740705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1712.795347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1712.841025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1712.878751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1712.896478] LNet: 88606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1712.915691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1712.968330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1713.011826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1713.038939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1713.073513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1713.103878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1713.565688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1713.626921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1713.705447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1713.748590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1713.788330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1713.826830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1713.893422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1713.925865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1713.975530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1713.999028] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1714.312889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1714.355992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1714.403612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1714.455716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1714.500957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1714.535854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.572602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.608490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.625587] LNet: 88726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.655887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.682972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.724982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.751885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.792121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.834815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.403972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1715.435453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1715.474305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1715.521484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1715.569268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1715.608323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1715.669932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1715.729373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1715.790534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1715.823521] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1716.148528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.190390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1716.220703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.251976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.276848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.298724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.322922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.347431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.362782] LNet: 88846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.376574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.400911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.445504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.487670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.527342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.561880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.098465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1717.150151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1717.187287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1717.245908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1717.305236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1717.345985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1717.389587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1717.443796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1717.514231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1717.830380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1717.881872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1717.907380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1717.963493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.003522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.038748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.070392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.115457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.161570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.220578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.261934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.290115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.313215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.345352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1718.875801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1718.913157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1718.944497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1719.001404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1719.034227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1719.084259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1719.131221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1719.171244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1719.251783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1719.270616] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1719.278394] LNet: Skipped 1 previous similar message [ 1719.595861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1719.626414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1719.680261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1719.728515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1719.753969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1719.778489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1719.827193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.879660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.917552] LNet: 89087:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.941784] LNet: 89087:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1719.960899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.992465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.021956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.056560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.093914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.137801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1720.715068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1720.780569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1720.828496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1720.883950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1720.920658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1720.981424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1721.058513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1721.112734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1721.157555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1721.188149] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1721.440876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1721.480955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1721.547871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1721.611566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1721.669914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1721.719289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1721.795636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1721.857685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1721.889992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1721.923688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1721.954804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.000988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.058279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.089661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1728.249862] LNet: 89281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.256318] LNetError: 89281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.269355] LNet: Removed LNI 192.168.203.45@tcp [ 1728.287268] LNet: Removed LNI 192.168.203.45@tcp1 [ 1728.889412] Key type .llcrypt unregistered [ 1728.894763] Key type ._llcrypt unregistered [ 1747.097491] Key type ._llcrypt registered [ 1747.102151] Key type .llcrypt registered [ 1748.416470] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:10:51 (1788264651) [ 1748.931223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1748.946880] alg: No test for adler32 (adler32-zlib) [ 1749.873693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.961172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.009400] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1750.798799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1750.834477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1750.859522] LNetError: 90258:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1750.943570] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1750.976971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.45@tcp [ 1751.079593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.347944] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:11:00 (1788264660) [ 1757.559462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.635087] LNet: 90847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.645652] LNetError: 90847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.670407] LNet: Removed LNI 192.168.203.45@tcp [ 1757.702237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.775145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.809863] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1758.665871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1758.686898] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1759.716449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.45@tcp [ 1759.738505] LNetError: 91114:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1759.784826] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1760.973168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.45@tcp [ 1768.854143] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:11:11 (1788264671) [ 1770.602514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg345-server [ 1772.617604] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:11:15 (1788264675) [ 1774.651417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg345-server [ 1776.713424] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:11:19 (1788264679) [ 1779.192434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg345-server [ 1781.053422] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:11:23 (1788264683) [ 1781.360818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.393365] LNet: 92736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.404764] LNetError: 92736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.417903] LNet: Removed LNI 192.168.203.45@tcp [ 1781.437075] LNet: Removed LNI 192.168.203.45@tcp1 [ 1781.467576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.537024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.563071] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1782.496115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1782.573242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1782.651686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1782.776043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1782.825871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1782.965267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1789.464660] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:11:32 (1788264692) [ 1789.675765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.712725] LNet: 93466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.736985] LNetError: 93466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.750253] LNetError: 93466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1789.803512] LNet: Removed LNI 192.168.203.45@tcp [ 1789.848372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.937387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1789.973930] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1797.773859] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:11:40 (1788264700) [ 1797.977892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.020958] LNet: 94168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.029921] LNetError: 94168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.085716] LNet: Removed LNI 192.168.203.45@tcp [ 1798.105874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.163846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.187247] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1805.624692] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:11:48 (1788264708) [ 1807.207964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1809.006829] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:11:51 (1788264711) [ 1809.664286] LNet: 95206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.678641] LNetError: 95206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.689796] LNet: Removed LNI 192.168.203.45@tcp [ 1810.480067] Key type .llcrypt unregistered [ 1810.486760] Key type ._llcrypt unregistered [ 1817.205422] Key type ._llcrypt registered [ 1817.207952] Key type .llcrypt registered [ 1817.489285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.511749] alg: No test for adler32 (adler32-zlib) [ 1818.515236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1818.583047] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1826.392982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1831.846141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.145@tcp [ 1832.307523] LNet: 95621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.314965] LNetError: 95621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.342650] LNet: Removed LNI 192.168.203.45@tcp [ 1832.850906] Key type .llcrypt unregistered [ 1832.852931] Key type ._llcrypt unregistered [ 1844.591406] Key type ._llcrypt registered [ 1844.593968] Key type .llcrypt registered [ 1845.700854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1847.443962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1849.516773] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:12:32 (1788264752) [ 1851.100313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1852.892676] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:12:35 (1788264755) [ 1855.124257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg345-server [ 1856.880756] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:12:39 (1788264759) [ 1858.844667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg345-server [ 1860.722833] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:12:43 (1788264763) [ 1862.651162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.676030] alg: No test for adler32 (adler32-zlib) [ 1863.922281] Lustre: Lustre: Build Version: 2.17.57_103_ga9ee299 [ 1864.272843] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1865.992453] Key type lgssc registered [ 1867.612913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1888.527977] Key type lgssc unregistered [ 1888.881764] LNet: 98804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.890357] LNetError: 98804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.903791] LNet: Removed LNI 192.168.203.45@tcp [ 1889.757833] Key type .llcrypt unregistered [ 1889.761609] Key type ._llcrypt unregistered [ 1905.264607] Key type ._llcrypt registered [ 1905.266839] Key type .llcrypt registered [ 1906.735312] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:13:29 (1788264809) [ 1907.112567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.125479] alg: No test for adler32 (adler32-zlib) [ 1908.065418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.192305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.260197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.324567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.374934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1914.637287] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:13:37 (1788264817) [ 1914.824457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.857544] LNet: 100215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.912115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.985943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.017102] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1915.999417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.021251] LNet: 100360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.025250] LNetError: 100360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.039228] LNet: Removed LNI 192.168.203.45@tcp [ 1916.073823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.123497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1916.151752] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1916.850164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.893714] LNetError: 100388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.909791] LNet: Removed LNI 192.168.203.45@tcp [ 1916.941463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.017561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1917.067297] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1925.133368] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:13:47 (1788264827) [ 1925.288842] LNet: 100971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.296592] LNet: 100971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1925.306386] LNetError: 100971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.335433] LNet: Removed LNI 192.168.203.45@tcp [ 1926.190914] Key type .llcrypt unregistered [ 1926.194806] Key type ._llcrypt unregistered [ 1935.216655] Key type ._llcrypt registered [ 1935.219909] Key type .llcrypt registered [ 1935.681810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1935.723649] alg: No test for adler32 (adler32-zlib) [ 1936.723281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.828504] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1936.841081] LNet: *** cfs_fail_loc=e003, val=100*** [ 1938.194690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1938.218453] LNet: *** cfs_fail_loc=e003, val=100*** [ 1938.223017] LNet: Skipped 2 previous similar messages [ 1938.227364] Ping reply received with mapping: [ 1938.230361] found metadata reply NI[192.168.203.45@tcp] [ 1938.240771] 2d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.246242] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.259812] 5a 5a 5a 5a 5a 5a 5a 5a [ 1938.610683] LNet: 101373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.627367] LNetError: 101373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.665259] LNet: Removed LNI 192.168.203.45@tcp [ 1939.395426] Key type .llcrypt unregistered [ 1939.397365] Key type ._llcrypt unregistered [ 1952.019843] Key type ._llcrypt registered [ 1952.029030] Key type .llcrypt registered [ 1954.525789] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:14:16 (1788264856) [ 1955.051570] Key type .llcrypt unregistered [ 1955.055076] Key type ._llcrypt unregistered [ 1962.035728] Key type ._llcrypt registered [ 1962.038590] Key type .llcrypt registered [ 1962.343308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.388655] alg: No test for adler32 (adler32-zlib) [ 1963.646484] LNetError: 102411:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1964.210476] Key type .llcrypt unregistered [ 1964.212481] Key type ._llcrypt unregistered [ 1971.567300] Key type ._llcrypt registered [ 1971.571489] Key type .llcrypt registered [ 1973.254651] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:14:35 (1788264875) [ 1974.248476] Key type .llcrypt unregistered [ 1974.260699] Key type ._llcrypt unregistered [ 1983.330838] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1983.348960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1986.063781] Key type ._llcrypt registered [ 1986.066590] Key type .llcrypt registered [ 1986.244961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.256937] alg: No test for adler32 (adler32-zlib) [ 1987.216470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.294905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1998.444824] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:15:00 (1788264900) [ 1998.593582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.656410] LNet: 104183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.675770] LNetError: 104183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1998.702649] LNet: Removed LNI 10.1.2.3@tcp [ 1998.757850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.905592] Key type .llcrypt unregistered [ 1999.913446] Key type ._llcrypt unregistered [ 2007.768778] Key type ._llcrypt registered [ 2007.775874] Key type .llcrypt registered [ 2008.080183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.104727] alg: No test for adler32 (adler32-zlib) [ 2009.020485] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2009.128224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2009.272924] LNet: 104475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.276958] LNetError: 104475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.290825] LNet: Removed LNI 192.168.203.45@tcp [ 2010.171556] Key type .llcrypt unregistered [ 2010.173379] Key type ._llcrypt unregistered [ 2022.711448] Key type ._llcrypt registered [ 2022.714459] Key type .llcrypt registered [ 2024.209346] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:15:26 (1788264926) [ 2025.014279] Key type .llcrypt unregistered [ 2025.020137] Key type ._llcrypt unregistered [ 2031.825645] Key type ._llcrypt registered [ 2031.830592] Key type .llcrypt registered [ 2032.108178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.134637] alg: No test for adler32 (adler32-zlib) [ 2040.804754] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:15:43 (1788264943) [ 2041.123252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.177512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.276944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.320261] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2042.160691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.197126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2042.262147] LNet: Removed LNI 192.168.203.45@tcp [ 2042.561123] LNet: 106215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.126510] Key type .llcrypt unregistered [ 2043.131786] Key type ._llcrypt unregistered [ 2055.579599] Key type ._llcrypt registered [ 2055.581246] Key type .llcrypt registered [ 2056.959827] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:15:59 (1788264959) [ 2057.530764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.555803] alg: No test for adler32 (adler32-zlib) [ 2058.580841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.668712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.731577] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2059.812113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2059.859446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.45@tcp [ 2059.877555] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2059.882635] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2059.886250] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2059, 0, 0 [ 2059.891105] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2059.894471] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2059.899049] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2059.903510] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2059, 0, 0 [ 2059.909094] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2059.915492] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2059.924158] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2059.937184] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2059, 0, 2109 [ 2059.953887] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2059.963229] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2059.972824] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2059.982161] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2059, 0, 2109 [ 2059.990817] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2067.879322] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:16:10 (1788264970) [ 2068.400743] LNet: 107752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.409625] LNetError: 107752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.433045] LNet: Removed LNI 192.168.203.45@tcp [ 2069.230539] Key type .llcrypt unregistered [ 2069.236464] Key type ._llcrypt unregistered [ 2076.632305] Key type ._llcrypt registered [ 2076.633916] Key type .llcrypt registered [ 2076.923479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2076.973960] alg: No test for adler32 (adler32-zlib) [ 2078.011907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2078.040889] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2086.016848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2095.278644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.351843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2095.381983] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 2096.514174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2096.644404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2096.657822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2096.666559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2099.179093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2099.202783] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2109.864596] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:16:52 (1788265012) [ 2110.064060] LNet: 109253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.077247] LNetError: 109253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.082301] LNet: Removed LNI 192.168.203.45@tcp [ 2111.135406] LNet: Removed LNI 192.168.203.45@tcp1 [ 2111.138489] LNetError: 109253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2111.150058] LNetError: 109253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2111.165232] LNet: Removed LNI 10.1.2.3@tcp99 [ 2111.761841] Key type .llcrypt unregistered [ 2111.766925] Key type ._llcrypt unregistered [ 2117.893638] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2117.899189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2120.493252] Key type ._llcrypt registered [ 2120.496538] Key type .llcrypt registered [ 2120.726687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.755624] alg: No test for adler32 (adler32-zlib) [ 2121.741044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.796453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2121.823680] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2122.439624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2122.466088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2124.540352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2124.582206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2124.611905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2124.626283] LNetError: 109998:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2124.650932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2124.686946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2125.008707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2125.097249] LNet: 110018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.104521] LNetError: 110018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2125.108170] LNet: Removed LNI 10.1.2.3@tcp [ 2126.118789] LNet: Removed LNI 192.168.203.45@tcp [ 2126.705503] Key type .llcrypt unregistered [ 2126.709412] Key type ._llcrypt unregistered [ 2137.792831] Key type ._llcrypt registered [ 2137.798138] Key type .llcrypt registered [ 2139.093179] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:17:21 (1788265041) [ 2139.437550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.454370] alg: No test for adler32 (adler32-zlib) [ 2140.393234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.494385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.520416] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2141.340330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2141.462876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg345-client@tcp [ 2148.266706] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:17:31 (1788265051) [ 2148.367204] LNet: 111550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.374972] LNetError: 111550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.405839] LNet: Removed LNI 192.168.203.45@tcp [ 2149.154648] Key type .llcrypt unregistered [ 2149.165335] Key type ._llcrypt unregistered [ 2156.682625] Key type ._llcrypt registered [ 2156.683915] Key type .llcrypt registered [ 2156.928596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.954675] alg: No test for adler32 (adler32-zlib) [ 2157.859250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2157.892806] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2157.916931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2164.202972] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:17:47 (1788265067) [ 2165.947662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2167.944856] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:17:50 (1788265070) [ 2169.601814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2171.495738] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:17:54 (1788265074) [ 2171.642164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.680844] LNet: 112976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.688676] LNetError: 112976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.722488] LNet: Removed LNI 192.168.203.45@tcp [ 2171.761835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.805407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.826431] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2172.516863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2172.617435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2179.737876] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:18:02 (1788265082) [ 2179.848835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.884513] LNet: 113686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.897792] LNetError: 113686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2179.942994] LNet: Removed LNI 192.168.203.45@tcp [ 2179.988630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.034623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2180.083726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2180.680383] Key type .llcrypt unregistered [ 2180.683188] Key type ._llcrypt unregistered [ 2191.831886] Key type ._llcrypt registered [ 2191.834424] Key type .llcrypt registered [ 2193.197681] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:18:15 (1788265095) [ 2193.695784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.715720] alg: No test for adler32 (adler32-zlib) [ 2194.734317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.819219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.840410] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2195.568834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2195.649381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2202.030915] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:18:25 (1788265105) [ 2202.108220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.124364] LNet: 115237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.129959] LNetError: 115237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.150111] LNet: Removed LNI 192.168.203.45@tcp [ 2202.184868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.231181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2202.252822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2209.077495] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:18:31 (1788265111) [ 2209.182544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.220087] LNet: 115815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.273056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.364919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2209.610760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2209.641537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2209.711149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2210.105690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2210.144781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2210.213205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.239286] LNet: 115856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.274402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.363701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2210.863931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2210.897716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2210.942837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2211.447037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2211.502342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2218.017547] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:18:40 (1788265120) [ 2218.161013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.190277] LNet: 116455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.228818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.280463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.307177] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2219.108445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2219.131275] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2219.969299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2220.043709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2220.271163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.320339] LNetError: 116736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.335175] LNet: Removed LNI 192.168.203.45@tcp [ 2220.344799] LNet: Removed LNI 192.168.203.45@tcp2 [ 2220.379792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.488946] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2220.511262] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2220.519938] LNet: 116742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.282239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2221.518322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.546463] LNetError: 116779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.550644] LNetError: 116779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2222.560203] LNet: Removed LNI 192.168.203.45@tcp [ 2222.573589] LNet: Removed LNI 192.168.203.45@tcp2 [ 2222.618755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.702368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.754435] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2223.493423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2223.516744] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2224.423063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2224.453222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2224.506317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2224.769952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.801363] LNet: 117064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.806822] LNet: 117064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2224.812953] LNetError: 117064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.817793] LNetError: 117064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2224.826498] LNet: Removed LNI 192.168.203.45@tcp [ 2225.894504] LNet: Removed LNI 192.168.203.45@tcp2 [ 2225.923609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.001521] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2226.007848] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2226.014336] LNet: 117070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2226.659900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2226.822822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.836905] LNet: Removed LNI 192.168.203.45@tcp [ 2227.878487] LNet: Removed LNI 192.168.203.45@tcp2 [ 2227.920248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.007488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.044706] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2228.840147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2228.872392] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2229.606816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2229.646913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2229.813638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.830499] LNetError: 117387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.835077] LNetError: 117387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2229.843673] LNet: Removed LNI 192.168.203.45@tcp [ 2229.852858] LNet: Removed LNI 192.168.203.45@tcp2 [ 2229.872368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.927302] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2229.933354] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2229.942688] LNet: 117393:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.292343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2230.509493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.583318] LNet: Removed LNI 192.168.203.45@tcp [ 2231.608447] LNet: Removed LNI 192.168.203.45@tcp2 [ 2231.635886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.702711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2231.734839] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2232.554597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2232.593520] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2233.634202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2233.682372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2233.739341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2234.001339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.039578] LNet: 117714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.055645] LNet: 117714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2235.105199] LNet: Removed LNI 192.168.203.45@tcp [ 2236.138067] LNet: Removed LNI 192.168.203.45@tcp2 [ 2236.202096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.400268] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2236.420311] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2236.426977] LNet: 117721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2237.173177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2237.466815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.487701] LNet: Removed LNI 192.168.203.45@tcp [ 2238.502897] LNet: Removed LNI 192.168.203.45@tcp2 [ 2238.536173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.651459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.675967] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2239.451232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2239.480649] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2240.363042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2240.425461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2240.496608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2240.917601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.962322] LNetError: 118055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.967795] LNetError: 118055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2240.974220] LNet: Removed LNI 192.168.203.45@tcp [ 2240.995644] LNet: Removed LNI 192.168.203.45@tcp2 [ 2241.057560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.190469] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2241.214302] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2241.230564] LNet: 118061:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2241.955434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2242.393053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.487135] LNet: Removed LNI 192.168.203.45@tcp [ 2243.497892] LNet: Removed LNI 192.168.203.45@tcp2 [ 2243.561249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.641953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2243.696541] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2244.637601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2244.667944] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2245.774203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2245.847215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2245.879636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2246.023647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2246.732846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.776447] LNet: Removed LNI 192.168.203.45@tcp [ 2248.869769] LNet: Removed LNI 192.168.203.45@tcp2 [ 2248.951889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.071680] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2249.075235] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2249.851618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2258.887782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2260.824569] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:19:23 (1788265163) [ 2260.949050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.978210] LNet: 119178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.983281] LNet: 119178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2261.005687] LNetError: 119178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.017182] LNetError: 119178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2261.030591] LNet: Removed LNI 192.168.203.45@tcp [ 2262.055039] LNet: Removed LNI 192.168.203.45@tcp2 [ 2262.099780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.466444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.515332] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2263.244688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2263.263654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2264.073921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.095635] LNet: Removed LNI 10.1.2.3@tcp [ 2264.105693] LNet: Removed LNI 192.168.203.45@tcp [ 2264.126471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.189534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2264.231800] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2264.237105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2264.869490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2264.906963] LNet: Removed LNI 192.168.203.45@tcp [ 2264.924794] LNet: Removed LNI 10.1.2.3@tcp [ 2271.940390] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:19:34 (1788265174) [ 2272.031333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.071044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.204526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.234247] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2273.013947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2273.071485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.115611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.164980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.213827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.276537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.332672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.370826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.433671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.499854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.553519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2273.744346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2280.570088] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:19:43 (1788265183) [ 2280.696211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.766631] LNet: Removed LNI 192.168.203.45@tcp [ 2281.807216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.915334] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2282.057954] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2282.071686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2282.089644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2283.465192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2283.492081] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2284.342699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2284.370532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2286.973737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2287.042356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2287.130338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.45@tcp -m PUT -e local_timeout -r 1 [ 2287.203778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2287.280676] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2287.287876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2287.613430] LNetError: 120853:0:(lib-move.c:4801:lnet_parse()) 192.168.203.45@tcp, src 192.168.203.45@tcp: Dropping PUT (error -108 looking up sender) [ 2288.674786] LNet: Removed LNI 10.1.2.3@tcp [ 2289.713534] LNet: Removed LNI 192.168.203.45@tcp [ 2290.187984] Key type .llcrypt unregistered [ 2290.191176] Key type ._llcrypt unregistered [ 2302.512656] Key type ._llcrypt registered [ 2302.515125] Key type .llcrypt registered [ 2303.652388] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:20:06 (1788265206) [ 2304.088619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.109674] alg: No test for adler32 (adler32-zlib) [ 2305.096873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.695094] LNet: 122164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.459101] Key type .llcrypt unregistered [ 2306.462768] Key type ._llcrypt unregistered [ 2314.473201] Key type ._llcrypt registered [ 2314.475063] Key type .llcrypt registered [ 2314.651441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2314.665330] alg: No test for adler32 (adler32-zlib) [ 2315.651854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2315.701734] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2323.046421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2323.276727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2323.885339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 2323.936523] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 2324.152789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2324.184082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2324.211325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2324.234663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2324.258969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2324.282793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2324.328258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2324.367447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2324.382763] LNet: 122611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.398761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2324.435765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2324.474965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2324.505520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2324.534197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2324.566642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2325.317794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2325.634405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2325.658705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2325.680339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2325.706526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2325.751226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2325.788713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2325.816515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2325.838471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2325.848340] LNet: 122728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2325.867349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2325.906617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2325.938171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2325.965354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2325.996348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2326.034730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2326.682403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 2326.921580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2326.957169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2326.990659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2327.021570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2327.043130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2327.066729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2327.094912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2327.118602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2327.136150] LNet: 122845:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2327.156801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2327.186337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2327.217152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2327.258875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2327.286504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2327.308145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2327.832609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2328.049721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2328.103599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2328.157960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2328.195775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2328.222757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2328.250564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2328.292873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2328.326906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2328.365916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2328.396955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2328.427906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2328.466596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2328.496237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2328.537734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2328.913812] LNet: 123008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.921141] LNetError: 123008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.938200] LNet: Removed LNI 192.168.203.45@tcp [ 2329.371294] Key type .llcrypt unregistered [ 2329.376377] Key type ._llcrypt unregistered [ 2345.916078] Key type ._llcrypt registered [ 2345.918608] Key type .llcrypt registered [ 2347.157363] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:20:49 (1788265249) [ 2347.586425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.611965] alg: No test for adler32 (adler32-zlib) [ 2348.539905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.919951] LNet: 123892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.525443] Key type .llcrypt unregistered [ 2349.529351] Key type ._llcrypt unregistered [ 2355.164565] Key type ._llcrypt registered [ 2355.175818] Key type .llcrypt registered [ 2355.422273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2355.440690] alg: No test for adler32 (adler32-zlib) [ 2356.362583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2356.388231] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2363.654992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2372.539871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.587947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2372.615216] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 2373.454103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2373.926203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 2374.137299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2374.181722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2374.212298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2374.242446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2374.266542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2374.289381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2374.316723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2374.341668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2374.362328] LNet: 124541:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.376407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2374.405985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2374.452149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2374.503548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2374.542206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2374.585835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2375.074630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2375.315665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2375.346622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2375.375457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2375.402890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2375.431364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.472315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.525242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.557398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2375.577634] LNet: 124658:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2375.605128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2375.652644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2375.683947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2375.712550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2375.737944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2375.788239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.490594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 2376.698900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2376.724374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2376.748342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2376.781572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2376.815362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2376.848710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2376.864520] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 2376.879918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2376.905624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2376.920642] LNet: 124776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.934601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2376.964760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2376.986858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2377.014317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2377.037210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2377.062664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2377.800591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2378.047507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2378.074639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2378.104924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2378.134929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2378.166753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2378.218743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2378.269219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2378.301056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2378.341573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2378.374627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2378.416131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2378.443349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2378.483350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2378.524427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2382.454773] LNet: 124973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.474769] LNetError: 124973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.489574] LNet: Removed LNI 192.168.203.45@tcp [ 2382.511763] LNet: Removed LNI 192.168.203.45@tcp1 [ 2382.885778] Key type .llcrypt unregistered [ 2382.896929] Key type ._llcrypt unregistered [ 2398.656546] Key type ._llcrypt registered [ 2398.659255] Key type .llcrypt registered [ 2399.925351] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:21:42 (1788265302) [ 2400.475669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.514271] alg: No test for adler32 (adler32-zlib) [ 2401.431745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.479473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2401.496438] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2402.153443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2402.175645] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2402.898808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2403.009920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2404.063240] LNet: Removed LNI 192.168.203.45@tcp [ 2411.020274] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:21:53 (1788265313) [ 2412.835488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2414.844919] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:21:57 (1788265317) [ 2422.247170] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:22:05 (1788265325) [ 2428.999554] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:22:11 (1788265331) [ 2435.380562] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:22:18 (1788265338) [ 2442.216215] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:22:25 (1788265345) [ 2449.300035] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:22:32 (1788265352) [ 2456.506632] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:22:39 (1788265359) [ 2464.313444] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:22:46 (1788265366) [ 2470.817849] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:22:53 (1788265373) [ 2477.192970] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:23:00 (1788265380) [ 2484.909604] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:23:07 (1788265387) [ 2493.617148] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:23:16 (1788265396) [ 2500.804571] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:23:23 (1788265403) [ 2507.489661] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:23:30 (1788265410) [ 2514.958938] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:23:37 (1788265417) [ 2522.930799] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:23:45 (1788265425) [ 2531.106085] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:23:53 (1788265433) [ 2538.956513] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:24:01 (1788265441) [ 2539.195613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2539.220110] LNet: 136663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.234322] LNetError: 136663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.250470] LNet: Removed LNI 192.168.203.45@tcp2 [ 2539.261677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.358811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2539.387648] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2540.683678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2540.723285] LNet: 136805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.738935] LNetError: 136805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.769036] LNet: Removed LNI 192.168.203.45@tcp [ 2540.802662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2540.896921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2540.940056] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2548.358406] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:24:11 (1788265451) [ 2555.629883] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:24:18 (1788265458) [ 2562.006175] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:24:25 (1788265465) [ 2568.471274] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:24:31 (1788265471) [ 2574.866330] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:24:37 (1788265477) [ 2582.436543] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:24:45 (1788265485) [ 2590.585236] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:24:53 (1788265493) [ 2598.006754] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:25:00 (1788265500) [ 2606.079633] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:25:08 (1788265508) [ 2614.558580] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:25:17 (1788265517) [ 2620.728361] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:25:23 (1788265523) [ 2626.695537] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:25:29 (1788265529) [ 2633.037343] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2420 sec ========== 08:25:36 (1788265536) [ 2633.604471] LNet: 144270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.616915] LNetError: 144270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.648036] LNet: Removed LNI 192.168.203.45@tcp [ 2634.691551] Key type .llcrypt unregistered [ 2634.701154] Key type ._llcrypt unregistered [ 2649.985609] Key type ._llcrypt registered [ 2649.992577] Key type .llcrypt registered [ 2650.880902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2650.927556] alg: No test for adler32 (adler32-zlib) [ 2652.116727] Lustre: Lustre: Build Version: 2.17.57_103_ga9ee299 [ 2652.536131] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2654.407200] Key type lgssc registered [ 2656.373206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2719.213761] Lustre: Mounted lustre-client - version 2.17.57_103_ga9ee299 [ 2723.718623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2730.866554] Lustre: Unmounted lustre-client [ 2758.647705] Key type lgssc unregistered [ 2758.868885] LNet: 147214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.879169] LNetError: 147214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.893364] LNet: Removed LNI 192.168.203.45@tcp [ 2759.552278] Key type .llcrypt unregistered [ 2759.557447] Key type ._llcrypt unregistered