[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442273512 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002466] x2apic enabled [ 0.003018] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009148] LSM: Security Framework initializing [ 0.010069] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.012114] *** VALIDATE selinux *** [ 0.022770] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028262] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032141] *** VALIDATE tmpfs *** [ 0.034521] *** VALIDATE proc *** [ 0.036070] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039106] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042028] Spectre V2 : User space: Vulnerable [ 0.043013] Speculative Store Bypass: Vulnerable [ 0.046370] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.048357] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049839] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050029] ... version: 2 [ 0.051010] ... bit width: 48 [ 0.052086] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054020] ... max period: 00007fffffffffff [ 0.055021] ... fixed-purpose events: 3 [ 0.056018] ... event mask: 000000070000000f [ 0.058277] rcu: Hierarchical SRCU implementation. [ 0.060848] smp: Bringing up secondary CPUs ... [ 0.061741] x86: Booting SMP configuration: [ 0.062034] .... node #0, CPUs: #1 #2 #3 [ 0.066484] smp: Brought up 1 node, 4 CPUs [ 0.068018] smpboot: Max logical packages: 1 [ 0.069023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.246945] node 0 deferred pages initialised in 175ms [ 0.250220] devtmpfs: initialized [ 0.251262] x86/mm: Memory block size: 128MB [ 0.253864] gcov: version magic: 0x41383552 [ 0.255665] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.259120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.262356] pinctrl core: initialized pinctrl subsystem [ 0.264266] [ 0.264839] ************************************************************* [ 0.266015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269016] ** ** [ 0.271017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274013] ** ** [ 0.276014] ** This means that this kernel is built to expose internal ** [ 0.278016] ** IOMMU data structures, which may compromise security on ** [ 0.281015] ** your system. ** [ 0.283014] ** ** [ 0.285015] ** If you see this message and you are not debugging the ** [ 0.286000] ** kernel, report this immediately to your vendor! ** [ 0.288014] ** ** [ 0.290011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292015] ************************************************************* [ 0.294812] NET: Registered protocol family 16 [ 0.296501] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.299136] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.302068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.306200] cpuidle: using governor menu [ 0.308068] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.310684] PCI: Using configuration type 1 for base access [ 0.313174] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324125] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326584] cryptd: max_cpu_qlen set to 1000 [ 0.328303] ACPI: Added _OSI(Module Device) [ 0.330018] ACPI: Added _OSI(Processor Device) [ 0.331037] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.338569] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343711] ACPI: Interpreter enabled [ 0.347275] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349038] ACPI: Using IOAPIC for interrupt routing [ 0.352185] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.356492] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.367556] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.369047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.372025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374161] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.378820] acpiphp: Slot [2] registered [ 0.381323] acpiphp: Slot [5] registered [ 0.383308] acpiphp: Slot [6] registered [ 0.385328] acpiphp: Slot [3] registered [ 0.387170] acpiphp: Slot [4] registered [ 0.389159] acpiphp: Slot [7] registered [ 0.390189] acpiphp: Slot [8] registered [ 0.392183] acpiphp: Slot [9] registered [ 0.394201] acpiphp: Slot [10] registered [ 0.396135] acpiphp: Slot [11] registered [ 0.397133] acpiphp: Slot [12] registered [ 0.398086] acpiphp: Slot [13] registered [ 0.399100] acpiphp: Slot [14] registered [ 0.401140] acpiphp: Slot [15] registered [ 0.402081] acpiphp: Slot [16] registered [ 0.403094] acpiphp: Slot [17] registered [ 0.404194] acpiphp: Slot [18] registered [ 0.406102] acpiphp: Slot [19] registered [ 0.407088] acpiphp: Slot [20] registered [ 0.408081] acpiphp: Slot [21] registered [ 0.409078] acpiphp: Slot [22] registered [ 0.410080] acpiphp: Slot [23] registered [ 0.411108] acpiphp: Slot [24] registered [ 0.412072] acpiphp: Slot [25] registered [ 0.413068] acpiphp: Slot [26] registered [ 0.414306] acpiphp: Slot [27] registered [ 0.416227] acpiphp: Slot [28] registered [ 0.418179] acpiphp: Slot [29] registered [ 0.420183] acpiphp: Slot [30] registered [ 0.422232] acpiphp: Slot [31] registered [ 0.424149] PCI host bridge to bus 0000:00 [ 0.426067] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.433041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.436039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.438039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.441041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.443239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447161] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450405] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.458544] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.462068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.468029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.475100] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.479082] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.482062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.486123] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493026] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.509019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.514743] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.521033] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.529021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.536942] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.554028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566825] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.573025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.602016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.615000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.618485] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.621394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.624446] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.626205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.631772] iommu: Default domain type: Passthrough [ 0.633441] SCSI subsystem initialized [ 0.635190] ACPI: bus type USB registered [ 0.637122] usbcore: registered new interface driver usbfs [ 0.639126] usbcore: registered new interface driver hub [ 0.641102] usbcore: registered new device driver usb [ 0.643205] pps_core: LinuxPPS API ver. 1 registered [ 0.646015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.649075] PTP clock support registered [ 0.651167] EDAC MC: Ver: 3.0.0 [ 0.653137] PCI: Using ACPI for IRQ routing [ 0.655798] NetLabel: Initializing [ 0.657012] NetLabel: domain hash size = 128 [ 0.658012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661122] NetLabel: unlabeled traffic allowed by default [ 0.663122] vgaarb: loaded [ 0.664353] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.666022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.671155] clocksource: Switched to clocksource kvm-clock [ 0.797792] VFS: Disk quotas dquot_6.6.0 [ 0.799445] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802846] *** VALIDATE ramfs *** [ 0.804339] *** VALIDATE hugetlbfs *** [ 0.807386] pnp: PnP ACPI init [ 0.810567] pnp: PnP ACPI: found 6 devices [ 0.836110] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838838] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840922] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843211] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845419] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847764] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.850462] NET: Registered protocol family 2 [ 0.852721] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857348] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860524] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.865667] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.868963] TCP: Hash tables configured (established 65536 bind 65536) [ 0.871710] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.874526] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877154] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880266] NET: Registered protocol family 1 [ 0.882739] RPC: Registered named UNIX socket transport module. [ 0.884547] RPC: Registered udp transport module. [ 0.885995] RPC: Registered tcp transport module. [ 0.887504] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889495] NET: Registered protocol family 44 [ 0.890956] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.892728] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.894611] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.896514] PCI: CLS 0 bytes, default 64 [ 0.897968] Unpacking initramfs... [ 2.322301] debug: unmapping init [mem 0xffff91467cc64000-0xffff91467ffcffff] [ 2.326565] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328966] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.332128] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.854519] Initialise system trusted keyrings [ 2.856747] Key type blacklist registered [ 2.858917] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.868924] zbud: loaded [ 2.872498] *** VALIDATE nfs *** [ 2.873951] *** VALIDATE nfs4 *** [ 2.875795] pstore: using deflate compression [ 2.882260] Platform Keyring initialized [ 2.989070] NET: Registered protocol family 38 [ 2.990657] Key type asymmetric registered [ 2.991977] Asymmetric key parser 'x509' registered [ 2.993723] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.996841] io scheduler mq-deadline registered [ 2.998554] io scheduler kyber registered [ 3.000396] io scheduler bfq registered [ 3.002289] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.005205] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.007614] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.009975] ACPI: Power Button [PWRF] [ 3.015465] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.022222] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.055255] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.084277] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.112780] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.118596] Non-volatile memory driver v1.3 [ 3.119914] Linux agpgart interface v0.103 [ 3.154218] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.159286] vda: detected capacity change from 0 to 74825728 [ 3.175960] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.180072] vdb: detected capacity change from 0 to 1073741824 [ 3.186440] libphy: Fixed MDIO Bus: probed [ 3.191686] usbcore: registered new interface driver usbserial_generic [ 3.194390] usbserial: USB Serial support registered for generic [ 3.196532] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.200858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.202933] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.205632] mousedev: PS/2 mouse device common for all mice [ 3.209089] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.210333] rtc_cmos 00:05: RTC can wake from S4 [ 3.217185] rtc_cmos 00:05: registered as rtc0 [ 3.219487] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.220642] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.223600] intel_pstate: CPU model not supported [ 3.229348] hid: raw HID events driver (C) Jiri Kosina [ 3.232895] usbcore: registered new interface driver usbhid [ 3.235163] usbhid: USB HID core driver [ 3.236987] drop_monitor: Initializing network drop monitor service [ 3.239776] Initializing XFRM netlink socket [ 3.239891] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.241851] NET: Registered protocol family 10 [ 3.249065] Segment Routing with IPv6 [ 3.250837] NET: Registered protocol family 17 [ 3.252835] mpls_gso: MPLS GSO support [ 3.259601] RAS: Correctable Errors collector initialized. [ 3.261846] AVX version of gcm_enc/dec engaged. [ 3.263884] AES CTR mode by8 optimization enabled [ 3.343839] sched_clock: Marking stable (3343678784, 0)->(4290596502, -946917718) [ 3.348840] registered taskstats version 1 [ 3.351176] Loading compiled-in X.509 certificates [ 3.353405] zswap: loaded using pool lzo/zbud [ 3.383667] Key type big_key registered [ 3.399097] Key type encrypted registered [ 3.400929] ima: No TPM chip found, activating TPM-bypass! [ 3.403309] ima: Allocated hash algorithm: sha1 [ 3.404644] ima: No architecture policies found [ 3.406389] evm: Initialising EVM extended attributes: [ 3.408662] evm: security.selinux [ 3.409748] evm: security.ima [ 3.410799] evm: security.capability [ 3.412796] evm: HMAC attrs: 0x1 [ 3.415200] rtc_cmos 00:05: setting system clock to 2026-08-24 23:18:31 UTC (1787613511) [ 3.422309] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 3.426029] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 3.440159] Write protecting the kernel read-only data: 28672k [ 3.444042] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 3.447145] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 3.485367] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.495554] systemd[1]: Detected virtualization kvm. [ 3.497548] systemd[1]: Detected architecture x86-64. [ 3.499161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.521763] systemd[1]: No hostname configured. [ 3.530548] systemd[1]: Set hostname to . [ 3.538084] random: systemd: uninitialized urandom read (16 bytes read) [ 3.540409] systemd[1]: Initializing machine ID from random generator. [ 3.591948] random: ln: uninitialized urandom read (6 bytes read) [ 3.682317] random: systemd: uninitialized urandom read (16 bytes read) [ 3.684728] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.689799] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.694662] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.368192] device-mapper: uevent: version 1.0.3 [ 4.370551] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.112848] virtio_net virtio0 ens2: renamed from eth0 [ 5.124052] random: fast init done [ 5.158598] scsi host0: ata_piix [ 5.163081] scsi host1: ata_piix [ 5.165174] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.167704] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.996886] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.918691] random: crng init done [ 9.922724] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.499981] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.712805] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.013634] SELinux: Disabled at runtime. [ 12.086788] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.095498] systemd[1]: Detected virtualization kvm. [ 12.097407] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.632118] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.635469] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.645696] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.650815] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.655611] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.663885] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.669355] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK [0[ 12.779524] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.158767] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.553852] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.623483] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.685302] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.697350] EDAC sbridge: Ver: 1.1.2 [ 15.095493] Key type dns_resolver registered [ 15.401182] NFS: Registering the id_resolver key type [ 15.403319] Key type id_resolver registered [ 15.404826] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-client login: [ 79.687622] libcfs: loading out-of-tree module taints kernel. [ 79.786480] Key type ._llcrypt registered [ 79.787792] Key type .llcrypt registered [ 80.259037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 80.274347] alg: No test for adler32 (adler32-zlib) [ 81.725257] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 82.604564] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 84.471365] Key type lgssc registered [ 87.127892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.172281] hrtimer: interrupt took 3746055 ns [ 245.888680] Lustre: Mounted lustre-client - version 2.17.57_80_g74d5152 [ 250.423709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.735634] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing check_logdir /tmp/testlogs/ [ 267.783769] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing yml_node [ 271.843895] Lustre: lustre-OST0000-osc-ffff9146c512e800: disconnect after 23s idle [ 272.026875] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 274.556522] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 277.125181] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 278.655221] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 19:23:05 EDT 2026 [ 294.255080] Lustre: Unmounted lustre-client [ 312.513329] Key type lgssc unregistered [ 312.803783] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.811526] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.840654] LNet: Removed LNI 192.168.204.60@tcp [ 313.859412] Key type .llcrypt unregistered [ 313.865861] Key type ._llcrypt unregistered [ 341.488270] Key type ._llcrypt registered [ 341.490224] Key type .llcrypt registered [ 341.725832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.756142] alg: No test for adler32 (adler32-zlib) [ 342.787946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 342.913443] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 342.955150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 348.161552] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 348.286934] LNet: 9301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.298040] LNetError: 9301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.327220] LNet: Removed LNI 192.168.204.60@tcp [ 349.149568] Key type .llcrypt unregistered [ 349.152452] Key type ._llcrypt unregistered [ 356.264728] Key type ._llcrypt registered [ 356.268171] Key type .llcrypt registered [ 357.597350] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:24:24 (1787613864) [ 357.985847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 358.020104] alg: No test for adler32 (adler32-zlib) [ 358.975511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.061904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.081584] LNet: 9721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.105891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.165328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.198297] LNet: 9728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 367.667777] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:24:33 (1787613873) [ 367.788610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.839496] LNet: 10301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.873141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.005865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 375.276489] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:24:41 (1787613881) [ 375.437544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.472796] LNet: 10889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.493619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.571177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 383.026379] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:24:49 (1787613889) [ 383.141477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.193403] LNet: 11472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.238442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.368025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 390.797972] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:24:57 (1787613897) [ 390.931712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.969680] LNet: 12056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.031237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.171460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 391.865300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 398.913934] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:25:05 (1787613905) [ 399.159664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.190309] LNet: 12654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.229229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.344727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 400.076242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 400.487152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 407.753089] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:25:13 (1787613913) [ 407.815465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.900908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.025184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 416.428987] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:25:22 (1787613922) [ 416.618665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.661123] LNet: 13829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.678478] LNet: 13829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 416.737698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.840807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 416.870454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 417.305262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 417.372807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 418.092476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 418.147975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.608362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 418.653868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.287862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 419.357313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.816314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 419.858349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.356360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 420.412630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.985852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 421.043597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 421.631233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 421.680917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.088604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 422.134676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 422.405047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 422.445886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.187971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 423.263900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 423.984500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 424.042280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 433.847154] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:25:40 (1787613940) [ 434.121942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.197882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.329219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 434.384162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 442.090952] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:25:48 (1787613948) [ 442.209449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.260953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.322672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 442.347287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 449.799395] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:25:56 (1787613956) [ 449.993996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.048982] LNet: 15685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.062746] LNet: 15685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 450.114720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.235939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 450.289906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 457.453407] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:26:04 (1787613964) [ 457.541756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.605944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.740646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.770179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 464.680760] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:26:11 (1787613971) [ 464.878734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.917964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.994096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 465.047675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 473.128836] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:26:19 (1787613979) [ 473.257469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.356427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.498317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 473.596750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 480.027642] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:26:26 (1787613986) [ 480.169063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.269677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.359651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.819452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.500691] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:26:34 (1787613994) [ 488.743190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.794871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.060360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.332535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.145294] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:26:42 (1787614002) [ 496.220803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.263313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.396764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.586105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.416430] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:26:50 (1787614010) [ 503.724373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.791267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.442874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.814933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.473811] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:26:58 (1787614018) [ 511.661810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.721473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.800709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.875696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.012168] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:27:05 (1787614025) [ 519.168190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.218944] LNet: 21108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.226461] LNet: 21108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 519.249489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.310481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.378620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.393143] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:27:13 (1787614033) [ 527.550568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.668764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.829073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 537.068965] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:27:22 (1787614042) [ 537.378282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.476442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.591655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 547.391782] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:27:33 (1787614053) [ 547.511716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.587500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.705952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 548.087546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 557.530670] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:27:44 (1787614064) [ 557.652702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.728696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.870776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 557.905968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 567.061547] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:27:52 (1787614072) [ 567.200972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.261509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.400119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 567.466892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 575.853593] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:28:02 (1787614082) [ 576.033169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.123370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.317228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 576.424788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 585.149814] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:28:11 (1787614091) [ 585.473807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.548682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.652805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 585.714252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 592.175875] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:28:18 (1787614098) [ 592.344367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.407610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.481700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 592.570187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 592.590832] LNetError: 25836:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 592.662073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 592.719565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 599.013200] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:28:25 (1787614105) [ 599.192487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.240661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.361134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MOVJstxvpd [ 599.445968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MOVJstxvpd [ 599.516118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MOVJstxvpd [ 607.158456] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:28:33 (1787614113) [ 607.272498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.333840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.479446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 607.522880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 607.570351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 614.613266] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:28:41 (1787614121) [ 614.861674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.894433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.996377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.043091] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 616.378706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 617.651853] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 619.560122] Key type lgssc registered [ 620.932568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.560744] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.60@tcp99 has no available nets [ 627.601742] Key type lgssc unregistered [ 628.129895] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.152879] LNet: Removed LNI 192.168.204.60@tcp [ 628.793374] Key type .llcrypt unregistered [ 628.797544] Key type ._llcrypt unregistered [ 636.256650] Key type ._llcrypt registered [ 636.258377] Key type .llcrypt registered [ 637.594393] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:29:04 (1787614144) [ 638.149457] Key type .llcrypt unregistered [ 638.158306] Key type ._llcrypt unregistered [ 645.173407] Key type ._llcrypt registered [ 645.176781] Key type .llcrypt registered [ 645.559598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.578786] alg: No test for adler32 (adler32-zlib) [ 646.591824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.748228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.792574] LNet: 29631:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 646.808727] LNet: 29631:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 646.816430] LNet: 29631:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 646.837212] LNet: Message forwarding will be enabled in 2s [ 648.929050] LNet: Message forwarding enabled [ 649.512106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 649.560875] LNet: Message forwarding will stop in 2s to 6s [ 653.045087] LNet: Message forwarding disabled [ 653.225608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 653.263445] LNet: 29711:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 653.270128] LNet: 29711:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 653.279020] LNet: 29711:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 653.294425] LNet: Message forwarding will be enabled in 2s [ 653.628560] LNet: 29724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.437607] Key type .llcrypt unregistered [ 654.440902] Key type ._llcrypt unregistered [ 666.508692] Key type ._llcrypt registered [ 666.513478] Key type .llcrypt registered [ 667.875785] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:29:34 (1787614174) [ 668.188849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.203329] alg: No test for adler32 (adler32-zlib) [ 669.115915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.193236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 675.496906] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:29:42 (1787614182) [ 675.652579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.697285] LNet: 31126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.732885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.823790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 675.867892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 675.956232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 675.994138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 676.032819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 676.065449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 676.116087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 676.152443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 676.200844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 676.242925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 676.289507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 676.329867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 676.376458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 676.432773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 676.478981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 676.534594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 676.590744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 676.661278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 676.728644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 676.752480] LNetError: 31176:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 682.729353] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:29:49 (1787614189) [ 682.963567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.996280] LNet: 31748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.020313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.086744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.692213] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:29:56 (1787614196) [ 691.759355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg460-server [ 693.803955] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:30:00 (1787614200) [ 694.127663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.148718] LNet: 32678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.170138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.225408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.250134] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 695.293602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[61-65]@tcp [ 695.308264] LNet: 32812:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 698.418137] LNetError: 32687:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.62@tcp has gone from up to down [ 701.544389] LNetError: 32814:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.65@tcp has gone from up to down [ 701.558783] LNetError: 32814:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 702.220119] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:30:08 (1787614208) [ 703.982949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg460-server [ 705.483852] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:30:12 (1787614212) [ 705.788853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.828844] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.851229] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.864871] LNet: Removed LNI 192.168.204.60@tcp [ 705.915378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.014869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.050222] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 706.861321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 706.886393] LNet: 33865:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 706.895413] LNet: 33865:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 706.930064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 712.987716] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:30:19 (1787614219) [ 713.143820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.195301] LNet: 34458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.201906] LNetError: 34458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.217762] LNet: Removed LNI 192.168.204.60@tcp [ 713.236711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.345138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 713.391713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.422920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.500666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.620925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.702488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.784149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 713.869713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.925724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.995491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.083251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.118587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.175958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 714.250458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.296279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.379112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.465574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.491805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.545231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 714.613518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.646717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.700417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.750832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.785721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.845918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.912420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.968095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.032260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.133275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.171940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.232755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 715.295249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.323801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.371568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 721.574384] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:30:28 (1787614228) [ 723.844930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg460-server [ 726.033770] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:30:32 (1787614232) [ 727.666545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg460-server [ 729.324943] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:30:36 (1787614236) [ 729.408089] LNet: 35850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.420525] LNet: 35850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 730.054640] Key type .llcrypt unregistered [ 730.062133] Key type ._llcrypt unregistered [ 737.241864] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 737.398645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 737.416326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 737.431618] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.082720] Key type ._llcrypt registered [ 740.085788] Key type .llcrypt registered [ 740.295276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.312691] alg: No test for adler32 (adler32-zlib) [ 741.248963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.303647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.339132] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 741.962453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.980766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 744.081470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 744.096699] LNet: Removed LNI 10.1.2.3@tcp [ 744.206759] LNet: 36582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.216344] LNetError: 36582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.228336] LNet: Removed LNI 192.168.204.60@tcp [ 744.878767] Key type .llcrypt unregistered [ 744.881214] Key type ._llcrypt unregistered [ 755.102271] Key type ._llcrypt registered [ 755.104751] Key type .llcrypt registered [ 756.409275] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:31:03 (1787614263) [ 756.895454] Key type .llcrypt unregistered [ 756.897360] Key type ._llcrypt unregistered [ 762.640968] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 762.789093] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.794878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.798801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 765.444750] Key type ._llcrypt registered [ 765.449633] Key type .llcrypt registered [ 765.732170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 765.748440] alg: No test for adler32 (adler32-zlib) [ 766.676906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.732675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.751219] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 767.464983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 767.498268] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 770.121683] LNet: 38083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.133960] LNetError: 38083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 770.141216] LNet: Removed LNI 10.1.2.3@tcp [ 770.167152] LNet: Removed LNI 192.168.204.60@tcp [ 770.970093] Key type .llcrypt unregistered [ 770.978819] Key type ._llcrypt unregistered [ 785.279725] Key type ._llcrypt registered [ 785.283160] Key type .llcrypt registered [ 786.687557] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:31:33 (1787614293) [ 787.411784] Key type .llcrypt unregistered [ 787.416743] Key type ._llcrypt unregistered [ 795.048203] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.220351] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.233205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.237047] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 797.834268] Key type ._llcrypt registered [ 797.839343] Key type .llcrypt registered [ 797.994262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.008201] alg: No test for adler32 (adler32-zlib) [ 799.144314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.244518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 799.323109] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 802.053305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.090902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.528815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 804.551376] LNet: Removed LNI 10.1.2.3@tcp [ 804.568211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 804.595436] LNet: Removed LNI 10.1.2.31@tcp [ 804.612859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 804.640070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.215718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 805.231606] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 806.270931] LNet: 40011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.278469] LNetError: 40011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 806.287230] LNet: Removed LNI 10.1.2.31@tcp [ 806.295667] LNet: Removed LNI 10.1.2.3@tcp [ 807.100573] Key type .llcrypt unregistered [ 807.102857] Key type ._llcrypt unregistered [ 818.902246] Key type ._llcrypt registered [ 818.907314] Key type .llcrypt registered [ 820.323722] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:32:06 (1787614326) [ 820.655779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 820.676912] alg: No test for adler32 (adler32-zlib) [ 821.616909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.675371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.701538] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 822.459896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.493041] LNet: 40967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.503544] LNetError: 40967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.525095] LNet: Removed LNI 192.168.204.60@tcp [ 822.561239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.635277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.60@tcp [ 822.667369] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 823.461746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.60@tcp [ 823.489090] LNet: Removed LNI 192.168.204.60@tcp [ 823.549891] LNet: 41102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.130116] Key type .llcrypt unregistered [ 824.135245] Key type ._llcrypt unregistered [ 835.339349] Key type ._llcrypt registered [ 835.342239] Key type .llcrypt registered [ 837.154678] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:32:23 (1787614343) [ 838.949542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg460-server [ 840.764460] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:32:27 (1787614347) [ 841.276558] Key type .llcrypt unregistered [ 841.286717] Key type ._llcrypt unregistered [ 848.667281] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.810581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.822429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.833198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.458716] Key type ._llcrypt registered [ 851.462571] Key type .llcrypt registered [ 851.706765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.729531] alg: No test for adler32 (adler32-zlib) [ 852.782315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.922945] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 853.771933] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 856.336968] LNet: 42943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.351257] LNetError: 42943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.360343] LNet: Removed LNI 192.168.204.60@tcp1 [ 856.375054] LNet: Removed LNI 10.1.2.3@tcp2 [ 857.299363] Key type .llcrypt unregistered [ 857.301208] Key type ._llcrypt unregistered [ 872.976616] Key type ._llcrypt registered [ 872.980774] Key type .llcrypt registered [ 874.452407] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:33:00 (1787614380) [ 875.047341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.058656] alg: No test for adler32 (adler32-zlib) [ 876.019189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.116493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.158756] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 876.975550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.996662] LNet: 43917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.001953] LNetError: 43917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.021525] LNet: Removed LNI 192.168.204.60@tcp [ 877.041691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.124806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.146699] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 883.541847] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:33:10 (1787614390) [ 883.782959] LNet: 44508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.800091] LNetError: 44508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.823922] LNet: Removed LNI 192.168.204.60@tcp [ 884.641586] Key type .llcrypt unregistered [ 884.645599] Key type ._llcrypt unregistered [ 891.799505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.991775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.999959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.020681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.776195] Key type ._llcrypt registered [ 894.777955] Key type .llcrypt registered [ 894.990156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.008553] alg: No test for adler32 (adler32-zlib) [ 895.922439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.001443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.035751] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 896.682168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 896.696281] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.854293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.896418] LNet: 45255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.908399] LNetError: 45255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.921684] LNet: Removed LNI 10.1.2.3@tcp [ 898.945421] LNet: Removed LNI 192.168.204.60@tcp [ 898.976210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.050406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.092957] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 899.102287] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.660572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.690874] LNet: 45282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.699771] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.708937] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 899.725132] LNet: Removed LNI 10.1.2.3@tcp [ 899.753712] LNet: Removed LNI 192.168.204.60@tcp [ 899.790984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.878961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.911745] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 899.916185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.435444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 900.461329] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 901.381762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.413906] LNet: 45445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.423362] LNetError: 45445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.437300] LNetError: 45445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.453094] LNet: Removed LNI 10.1.2.3@tcp [ 901.454816] LNet: Removed LNI 192.168.204.60@tcp [ 901.476772] LNet: Removed LNI 192.168.204.60@tcp2 [ 901.514908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.614228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.648668] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 901.657181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.665576] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 909.572673] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:33:36 (1787614416) [ 909.735573] LNet: 46035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.748128] LNetError: 46035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.758575] LNetError: 46035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 909.771162] LNet: Removed LNI 10.1.2.3@tcp [ 909.776190] LNet: Removed LNI 192.168.204.60@tcp [ 909.804093] LNet: Removed LNI 192.168.204.60@tcp2 [ 910.609740] Key type .llcrypt unregistered [ 910.611474] Key type ._llcrypt unregistered [ 917.653706] Key type ._llcrypt registered [ 917.660642] Key type .llcrypt registered [ 917.935335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.960632] alg: No test for adler32 (adler32-zlib) [ 918.948091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.023195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.066322] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 920.013329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.034249] LNet: 46442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.042993] LNetError: 46442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.059506] LNet: Removed LNI 192.168.204.60@tcp [ 920.076611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.122372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 920.164875] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 920.809239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.825276] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.834776] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.849269] LNet: Removed LNI 192.168.204.60@tcp [ 920.889349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.971195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 920.997841] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 921.606611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.663215] LNet: Removed LNI 192.168.204.60@tcp [ 921.699888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.803611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 921.857176] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 922.458096] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 922.728294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 922.739713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 922.750750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 925.121654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.148557] LNet: 46581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.155464] LNet: 46581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 925.161342] LNetError: 46581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.165897] LNetError: 46581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 925.178579] LNet: Removed LNI 192.168.204.60@tcp [ 925.203166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.292822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 925.326735] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 932.136647] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:33:59 (1787614439) [ 932.269577] LNet: 47174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.281840] LNetError: 47174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.311231] LNet: Removed LNI 192.168.204.60@tcp [ 932.971874] Key type .llcrypt unregistered [ 932.976971] Key type ._llcrypt unregistered [ 940.149314] Key type ._llcrypt registered [ 940.153611] Key type .llcrypt registered [ 940.453256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.494267] alg: No test for adler32 (adler32-zlib) [ 941.368942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.456122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.492881] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 942.829752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.858740] LNet: 47581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.865773] LNetError: 47581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.881503] LNet: Removed LNI 192.168.204.60@tcp [ 942.900519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.946549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 942.966859] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 949.991139] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:34:16 (1787614456) [ 950.240637] LNet: 48173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.251272] LNetError: 48173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.272870] LNet: Removed LNI 192.168.204.60@tcp [ 950.968901] Key type .llcrypt unregistered [ 950.972587] Key type ._llcrypt unregistered [ 957.171692] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.398760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.406196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.429519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.886309] Key type ._llcrypt registered [ 959.887915] Key type .llcrypt registered [ 960.011948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.028065] alg: No test for adler32 (adler32-zlib) [ 960.910095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.972740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.022606] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 961.972356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.017286] LNet: 48626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.023568] LNetError: 48626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.048634] LNet: Removed LNI 192.168.204.60@tcp [ 962.078957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.179182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 962.223295] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 970.331623] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:34:36 (1787614476) [ 970.459106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.509702] LNet: 49219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.518987] LNetError: 49219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.538748] LNet: Removed LNI 192.168.204.60@tcp [ 970.578687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.739259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 977.879440] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:34:44 (1787614484) [ 978.138903] LNet: 49805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.758413] Key type .llcrypt unregistered [ 978.760068] Key type ._llcrypt unregistered [ 986.365531] Key type ._llcrypt registered [ 986.369597] Key type .llcrypt registered [ 986.580851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 986.604859] alg: No test for adler32 (adler32-zlib) [ 987.484286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.525678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.559223] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 988.225549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.249769] LNet: 50210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.258862] LNetError: 50210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.270102] LNet: Removed LNI 192.168.204.60@tcp [ 988.298837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.390849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 988.426682] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 993.926949] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:35:00 (1787614500) [ 994.290255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.317357] LNet: 50803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.325770] LNetError: 50803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.346719] LNet: Removed LNI 192.168.204.60@tcp [ 994.393507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.450918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1000.126473] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:35:06 (1787614506) [ 1001.696131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1003.178646] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:35:10 (1787614510) [ 1004.546709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1006.005833] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:35:12 (1787614512) [ 1007.288891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1008.613910] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:35:15 (1787614515) [ 1009.887393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1011.406488] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:35:18 (1787614518) [ 1012.755331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1014.076332] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:35:21 (1787614521) [ 1014.199806] LNet: 52853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.778536] Key type .llcrypt unregistered [ 1014.781297] Key type ._llcrypt unregistered [ 1020.102729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.218594] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.225376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.234200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.812694] Key type ._llcrypt registered [ 1022.815809] Key type .llcrypt registered [ 1022.996597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.012420] alg: No test for adler32 (adler32-zlib) [ 1023.992868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.091162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.110987] LNet: Added LNI 192.168.204.60@tcp [8/512/0/180] [ 1024.117708] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1024.120649] LNet: 53183:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.210928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.228434] LNet: 53202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.234592] LNetError: 53202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.239781] LNet: Removed LNI 10.1.2.3@tcp [ 1024.242071] LNetError: 53202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.262490] LNet: Removed LNI 192.168.204.60@tcp [ 1024.280881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.412018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.445157] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1024.536344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.574072] LNet: 53226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.581226] LNetError: 53226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.596964] LNet: Removed LNI 192.168.204.60@tcp [ 1024.641299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.719355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1024.755460] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1024.759910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.762993] LNet: 53234:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1024.877265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.925216] LNet: 53252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.936700] LNetError: 53252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.951751] LNet: Removed LNI 10.1.2.3@tcp [ 1024.957184] LNetError: 53252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.982097] LNet: Removed LNI 192.168.204.60@tcp [ 1025.015044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.073422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1025.090291] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1025.095090] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.097604] LNet: 53261:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1025.257832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.285707] LNet: 53287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.292096] LNetError: 53287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.298241] LNet: Removed LNI 10.1.2.3@tcp [ 1025.304278] LNetError: 53287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.331796] LNet: Removed LNI 192.168.204.60@tcp [ 1025.354041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.438872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1025.463188] LNet: Added LNI 192.168.204.60@tcp [8/256/0/182] [ 1025.472861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1025.581850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.602744] LNet: 53315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.608684] LNetError: 53315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.620368] LNet: Removed LNI 10.1.2.3@tcp [ 1025.624233] LNetError: 53315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.638495] LNet: Removed LNI 192.168.204.60@tcp [ 1025.658931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.735527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1025.754501] LNet: Added LNI 192.168.204.60@tcp [8/256/0/182] [ 1025.759133] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1025.761980] LNet: 53323:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1025.843240] LNet: Removed LNI 10.1.2.3@tcp [ 1025.852474] LNet: Removed LNI 192.168.204.60@tcp [ 1026.475946] Key type .llcrypt unregistered [ 1026.480151] Key type ._llcrypt unregistered [ 1036.888317] Key type ._llcrypt registered [ 1036.894670] Key type .llcrypt registered [ 1038.268957] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:35:44 (1787614544) [ 1039.487497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1041.347072] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:35:48 (1787614548) [ 1041.698851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.714596] alg: No test for adler32 (adler32-zlib) [ 1042.613041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.675740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1042.719644] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1043.465611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.60 [ 1043.492335] LNet: 54591:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1043.504402] LNet: Removed LNI 192.168.204.60@tcp [ 1049.500245] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:35:56 (1787614556) [ 1049.588766] LNet: 55153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.600842] LNetError: 55153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.616528] LNet: Removed LNI 192.168.204.60@tcp [ 1050.090416] Key type .llcrypt unregistered [ 1050.095814] Key type ._llcrypt unregistered [ 1055.348293] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.648235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.395106] Key type ._llcrypt registered [ 1058.396736] Key type .llcrypt registered [ 1058.593220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.605972] alg: No test for adler32 (adler32-zlib) [ 1060.000671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1066.095776] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:36:12 (1787614572) [ 1066.379271] LNet: 56077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.404674] LNetError: 56077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.423743] LNet: Removed LNI 10.1.2.3@tcp [ 1067.106878] Key type .llcrypt unregistered [ 1067.112685] Key type ._llcrypt unregistered [ 1073.366039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1073.686774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1073.693878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1073.702841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1076.533869] Key type ._llcrypt registered [ 1076.535981] Key type .llcrypt registered [ 1076.723852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.746963] alg: No test for adler32 (adler32-zlib) [ 1077.689498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1084.510402] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:36:30 (1787614590) [ 1086.630616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1088.435405] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:36:35 (1787614595) [ 1088.776140] LNet: 57265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.780623] LNetError: 57265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1088.799521] LNet: Removed LNI 10.1.2.3@tcp [ 1089.828204] Key type .llcrypt unregistered [ 1089.834136] Key type ._llcrypt unregistered [ 1100.160337] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1100.667701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1100.693957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1100.709768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1103.983789] Key type ._llcrypt registered [ 1103.990650] Key type .llcrypt registered [ 1104.397773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.438285] alg: No test for adler32 (adler32-zlib) [ 1105.708337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.825938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1115.407417] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:37:01 (1787614621) [ 1116.062856] LNet: 58500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.078440] LNetError: 58500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.093893] LNet: Removed LNI 10.1.2.3@tcp [ 1117.187683] Key type .llcrypt unregistered [ 1117.195926] Key type ._llcrypt unregistered [ 1123.309728] Key type ._llcrypt registered [ 1123.312767] Key type .llcrypt registered [ 1123.452111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.470074] alg: No test for adler32 (adler32-zlib) [ 1124.348226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.375868] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1131.134636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1131.735711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1131.785589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1131.835862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1131.863259] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1132.080788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.113776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.146836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.172798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.200993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.231766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.257807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.288909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.305773] LNet: 58941:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.359594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.414566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.452295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.508707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.547495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.576737] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1132.579511] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1132.605051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.653836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1133.133233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1133.172382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1133.203541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1133.440791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.467986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.490445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.520305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.549796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.584720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.613412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.635667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.649919] LNet: 59050:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.670428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.701395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.731859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.766427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.798810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.829835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.866369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.356723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1134.414277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1134.451324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1134.469314] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1134.475892] LNet: Skipped 1 previous similar message [ 1134.687622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.717979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.745680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.779887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.820088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.854357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.896466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.934241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.948909] LNet: 59159:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1134.969901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.997187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.027635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.072855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.093934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.126536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.161763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1135.661586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1135.710747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1135.741914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1136.028063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.051112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.097880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.139098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.184332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.226129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.275918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.319786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.359810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.400568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.427945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.456448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.487770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.534124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.575932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.987456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1137.013809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1137.044826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1137.058971] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1137.071795] LNet: Skipped 2 previous similar messages [ 1137.243633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.271175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.306176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.347340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.379147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.406637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.458902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.514827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.546696] LNet: 59378:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.553192] LNet: 59378:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1137.568119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.608877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.642968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.667378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.704407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.740888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.781940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.237723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1138.283623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1138.329969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1138.565580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.593409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.635965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.669566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.694141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.720609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.760891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.792993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.829117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.864415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.895141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.923627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.955791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.981262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.006544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.218083] LNet: 59532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.223279] LNetError: 59532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.240402] LNet: Removed LNI 192.168.204.60@tcp [ 1139.596549] Key type .llcrypt unregistered [ 1139.599788] Key type ._llcrypt unregistered [ 1152.613852] Key type ._llcrypt registered [ 1152.617959] Key type .llcrypt registered [ 1153.775359] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:37:40 (1787614660) [ 1154.475487] Key type .llcrypt unregistered [ 1154.481896] Key type ._llcrypt unregistered [ 1160.378470] Key type ._llcrypt registered [ 1160.380815] Key type .llcrypt registered [ 1160.669200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.683715] alg: No test for adler32 (adler32-zlib) [ 1161.588245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.614088] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1169.335255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1179.812044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1179.846775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.870307] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1181.102869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1181.129280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 1181.162236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1181.196948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 1181.239887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1181.283669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 1181.308641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1181.339935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 1181.380486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1181.409234] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1181.414648] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1181.660689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.698308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.740154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.780588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.821583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.854870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.890472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.953086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.988732] LNet: 61003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.012707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.071188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.109466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.133402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.171040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.207426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.249292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.738730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1182.779358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 1182.809273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1182.836218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 1182.872959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1182.920088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 1182.958070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1182.991259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 1183.048809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1183.077210] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1183.085670] LNet: Skipped 2 previous similar messages [ 1183.281705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.322691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.370370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.395990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.441508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.479629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.520386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.576905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.598428] LNet: 61124:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.626534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.680944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.727276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.763906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.808802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.839740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.869990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.371903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1184.404690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 1184.429827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1184.455532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 1184.497833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1184.538641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 1184.584094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1184.628805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 1184.674063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1184.698660] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1184.706560] LNet: Skipped 2 previous similar messages [ 1184.902428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.934262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.959975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.982491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.004530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.028649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.051489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.083321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.104534] LNet: 61246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.123534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.164359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.210762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.247221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.280534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.327427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.363709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.880987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1185.913753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 1185.950661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1185.984652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 1186.014844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1186.055956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 1186.085132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1186.109316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 1186.154302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1186.351806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.396908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.429917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.457668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.480980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.506946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.529396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.557708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.600252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.637929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.675413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.716678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.766480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.811531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.851244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.300495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1187.334974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 1187.367246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1187.397878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 1187.428272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1187.460646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 1187.494104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1187.528442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 1187.572708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1187.600797] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1187.608852] LNet: Skipped 5 previous similar messages [ 1187.831598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.877125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.903647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.939259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.973133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.008628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.042169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.073598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.094117] LNet: 61488:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.103940] LNet: 61488:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1188.120356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.160894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.198814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.219685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.264863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.299142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.341432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.873887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1188.905989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 1188.938933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1188.971155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 1189.005460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1189.047797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 1189.089851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1189.118224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 1189.151793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1189.321464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.367120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.406311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.437045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.477962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.515702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.565141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.600461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.634191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.682384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.737747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.788575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.837752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.866993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.893986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1194.543833] LNet: 61689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.549472] LNetError: 61689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.555970] LNet: Removed LNI 192.168.204.60@tcp [ 1194.566634] LNet: Removed LNI 192.168.204.60@tcp1 [ 1194.914087] Key type .llcrypt unregistered [ 1194.920851] Key type ._llcrypt unregistered [ 1210.322774] Key type ._llcrypt registered [ 1210.325984] Key type .llcrypt registered [ 1211.501882] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:38:38 (1787614718) [ 1212.325283] Key type .llcrypt unregistered [ 1212.334080] Key type ._llcrypt unregistered [ 1217.867029] Key type ._llcrypt registered [ 1217.869276] Key type .llcrypt registered [ 1218.122560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.138022] alg: No test for adler32 (adler32-zlib) [ 1219.042616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.085539] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1226.001265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1226.657700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1226.683556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1226.716520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1226.734821] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1227.012208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.055298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.109977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.147297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.187887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.235970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.292206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.342709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.377730] LNet: 62940:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.401784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.439199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.507351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.547229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.588466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.648025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.702734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.321800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1228.359320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1228.429578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1228.459830] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1228.673709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.703078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.741730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.780356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.807929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.835954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.871209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.923215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.941876] LNet: 63049:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.969540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.030303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.084142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.117876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.171851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.274856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.339487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.905139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1229.955130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1230.013287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1230.043566] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1230.314824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.360415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.419342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.455889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.495452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.536326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.574637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.627073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.661271] LNet: 63158:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.694039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.740207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.771365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.798296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.846786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.899203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.938916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.330242] LNet: 63204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.340039] LNetError: 63204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.352120] LNet: Removed LNI 192.168.204.60@tcp [ 1231.826509] Key type .llcrypt unregistered [ 1231.829939] Key type ._llcrypt unregistered [ 1245.416962] Key type ._llcrypt registered [ 1245.421650] Key type .llcrypt registered [ 1246.537400] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:39:13 (1787614753) [ 1247.188606] Key type .llcrypt unregistered [ 1247.196498] Key type ._llcrypt unregistered [ 1252.224940] Key type ._llcrypt registered [ 1252.226854] Key type .llcrypt registered [ 1252.372690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1252.383430] alg: No test for adler32 (adler32-zlib) [ 1253.250177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1253.290596] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1259.605765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1266.993913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.034190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.059847] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1268.233410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1268.269605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1268.301151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1268.329540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1268.363539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1268.392237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1268.423166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1268.453597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1268.480286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1268.495507] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1268.502508] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1268.718327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.751473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.786962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.814148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.856658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.899185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.927596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.972309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.011778] LNet: 64672:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.054884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.094169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.135928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.165689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.207218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.244216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.271958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1269.314602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.824945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1269.849455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1269.873591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1269.898641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1269.934612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1269.966646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1270.005930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1270.039326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1270.067925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1270.081555] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1270.089253] LNet: Skipped 2 previous similar messages [ 1270.267297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.296946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.325397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.348204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.373909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.397610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.423097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.451696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.478233] LNet: 64796:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.494909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.522356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.559275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.588113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.616380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.663181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.722962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.775415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.242454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1271.270321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1271.295896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1271.321036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1271.347333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1271.371768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1271.395641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1271.422428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1271.450301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1271.468397] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1271.685441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.712967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1271.737678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.786477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.831646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.853927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.883607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.916689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.935566] LNet: 64919:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.957185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.980404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.001708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.019559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.045647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.082982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.121363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1272.150407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.629846] LNet: 65000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.640559] LNetError: 65000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.650908] LNet: Removed LNI 192.168.204.60@tcp [ 1276.659473] LNet: Removed LNI 192.168.204.60@tcp1 [ 1277.080889] Key type .llcrypt unregistered [ 1277.086151] Key type ._llcrypt unregistered [ 1292.244973] Key type ._llcrypt registered [ 1292.247908] Key type .llcrypt registered [ 1293.339568] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:40:00 (1787614800) [ 1294.726810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1296.395433] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:40:03 (1787614803) [ 1297.160331] Key type .llcrypt unregistered [ 1297.165221] Key type ._llcrypt unregistered [ 1303.568552] Key type ._llcrypt registered [ 1303.571604] Key type .llcrypt registered [ 1303.770258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1303.788290] alg: No test for adler32 (adler32-zlib) [ 1304.764382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1304.794343] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1312.144245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1312.755956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1312.793551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1312.836101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1312.862156] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1312.992903] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1313.126215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1313.162512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1313.191379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1313.218391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1313.254247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.282785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.309895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.344352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.367417] LNet: 66547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1313.385729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.413074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.451439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.478602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.516789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.546655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1313.815777] LNet: 66590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.836311] LNetError: 66590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.859707] LNet: Removed LNI 192.168.204.60@tcp [ 1314.233661] Key type .llcrypt unregistered [ 1314.241828] Key type ._llcrypt unregistered [ 1328.589212] Key type ._llcrypt registered [ 1328.591729] Key type .llcrypt registered [ 1328.735486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.748531] alg: No test for adler32 (adler32-zlib) [ 1329.644885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.697586] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1335.534821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1344.425249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.468905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.517965] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1345.732042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1345.765598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1345.798496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1345.832327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1345.867502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1345.896516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1345.944614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1345.989794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1346.044302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1346.067476] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1346.082105] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1346.084777] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1346.258982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1346.291081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1346.326091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1346.363447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1346.405393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.460560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.515704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.550593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.574729] LNet: 67472:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.603105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.629590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.672612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.698861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.742934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1346.781566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.825838] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1351.138108] LNet: 67549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.153985] LNetError: 67549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.164111] LNet: Removed LNI 192.168.204.60@tcp [ 1351.183797] LNet: Removed LNI 192.168.204.60@tcp1 [ 1351.607489] Key type .llcrypt unregistered [ 1351.613110] Key type ._llcrypt unregistered [ 1366.412893] Key type ._llcrypt registered [ 1366.414560] Key type .llcrypt registered [ 1367.675749] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:41:14 (1787614874) [ 1368.010781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.029238] alg: No test for adler32 (adler32-zlib) [ 1369.030221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.099333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.148971] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1370.170369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.188177] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1371.119235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1371.176630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1371.224300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1371.256739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1371.282930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1371.308421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1371.330015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1371.353457] LNetError: 68650:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1372.127310] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1372.127871] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1374.175419] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1374.806961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.277259] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1378.577382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.616784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.688482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.721409] LNet: 68758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.731990] LNetError: 68758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.736635] LNet: Removed LNI 192.168.204.60@tcp [ 1378.749835] LNet: Removed LNI 192.168.204.60@tcp1 [ 1378.765840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.812947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.839268] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1379.537944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.561275] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1380.532021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1380.604927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.674340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1380.732817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1380.781692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1380.816934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1380.866353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1380.899546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1380.925311] LNetError: 69030:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1381.855523] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1384.748859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1387.999471] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1388.004151] LNet: Skipped 1 previous similar message [ 1391.695304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1391.742585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1391.792956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1398.711218] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:41:45 (1787614905) [ 1398.827920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1398.855625] LNet: 69732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.861987] LNetError: 69732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.871754] LNetError: 69732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1398.877653] LNet: Removed LNI 192.168.204.60@tcp [ 1398.896316] LNet: Removed LNI 192.168.204.60@tcp1 [ 1398.944100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.021399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.062255] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1399.720884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1399.739878] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1400.616633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1400.692894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1400.731710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1400.767886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1400.822660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1400.874449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1400.913647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1401.029989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1401.055450] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.60@tcp [ 1401.058815] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1401.070173] LNet: Skipped 1 previous similar message [ 1416.310600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1416.468579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1416.500720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1416.529327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 500 [ 1416.651829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1423.474299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1423.556928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.579895] LNet: 70229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.587284] LNetError: 70229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.595592] LNetError: 70229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.601603] LNet: Removed LNI 192.168.204.60@tcp [ 1423.611140] LNet: Removed LNI 192.168.204.60@tcp1 [ 1423.630462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.740236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.774259] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1424.431542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1424.447072] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1425.209641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1425.281864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1425.391944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1425.433518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1425.476726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1425.517151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1425.551630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1425.605209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1426.788498] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1426.793698] LNet: Skipped 3 previous similar messages [ 1426.950807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.413602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.445990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1437.482186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.054370] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:42:31 (1787614951) [ 1445.675259] LNet: 71275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.685483] LNetError: 71275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.690770] LNetError: 71275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1446.751131] LNet: Removed LNI 192.168.204.60@tcp [ 1446.770538] LNet: Removed LNI 192.168.204.60@tcp1 [ 1447.799642] Key type .llcrypt unregistered [ 1447.805717] Key type ._llcrypt unregistered [ 1455.655642] Key type ._llcrypt registered [ 1455.665753] Key type .llcrypt registered [ 1456.000768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1456.025511] alg: No test for adler32 (adler32-zlib) [ 1456.955152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1456.990092] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1467.539236] Lustre: DEBUG MARKER: Initial discovery [ 1467.756556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1470.023724] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1470.242831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1470.300276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1470.339765] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1470.369414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1470.397764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -l 3 [ 1470.434308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -e local_error [ 1470.478242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m PUT -l 6 [ 1471.327399] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1472.089485] Lustre: DEBUG MARKER: Force oleg460-client.virtnet to discover 192.168.204.160@tcp (in background) [ 1472.174171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1473.703323] Lustre: DEBUG MARKER: Force oleg460-server to discover 192.168.204.60@tcp [ 1475.234031] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1476.089789] Lustre: DEBUG MARKER: Wait for 72038 [ 1477.702670] Lustre: DEBUG MARKER: Finished wait on 72038 [ 1477.830802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1477.860566] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1477.870354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1478.158377] LNet: 72498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.167913] LNetError: 72498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.184696] LNet: Removed LNI 192.168.204.60@tcp [ 1478.620723] Key type .llcrypt unregistered [ 1478.623058] Key type ._llcrypt unregistered [ 1495.292566] Key type ._llcrypt registered [ 1495.297257] Key type .llcrypt registered [ 1496.797440] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:43:23 (1787615003) [ 1497.356424] Key type .llcrypt unregistered [ 1497.360843] Key type ._llcrypt unregistered [ 1504.809658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1505.002974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1505.016366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1505.023574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1507.669596] Key type ._llcrypt registered [ 1507.674759] Key type .llcrypt registered [ 1507.911786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.938386] alg: No test for adler32 (adler32-zlib) [ 1508.972222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.032521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.065567] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1510.068431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1510.105174] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1518.496352] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:43:45 (1787615025) [ 1518.815154] LNet: 74579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.834838] LNetError: 74579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1518.844123] LNet: Removed LNI 10.1.2.3@tcp [ 1518.859143] LNet: Removed LNI 192.168.204.60@tcp [ 1519.511676] Key type .llcrypt unregistered [ 1519.517411] Key type ._llcrypt unregistered [ 1526.345396] Key type ._llcrypt registered [ 1526.350354] Key type .llcrypt registered [ 1526.607441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1526.620985] alg: No test for adler32 (adler32-zlib) [ 1527.520563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.581456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.611545] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1528.317612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1528.338375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1529.254214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.60@tcp [ 1530.338426] LNetError: Unexpected error -22 connecting to 192.168.204.60@tcp at host 192.168.204.60:988 [ 1535.504321] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:44:02 (1787615042) [ 1535.827715] LNet: 75715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.836344] LNetError: 75715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1535.845863] LNet: Removed LNI 10.1.2.3@tcp [ 1535.861076] LNet: Removed LNI 192.168.204.60@tcp [ 1536.631549] Key type .llcrypt unregistered [ 1536.634650] Key type ._llcrypt unregistered [ 1543.481424] Key type ._llcrypt registered [ 1543.483532] Key type .llcrypt registered [ 1543.694801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1543.715630] alg: No test for adler32 (adler32-zlib) [ 1544.670957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.735400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1544.778924] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1545.718109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1545.748590] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1546.829365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.60@tcp1 --nid 192.168.204.60@tcp2 [ 1551.228857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp2 [ 1551.256672] LNetError: 76555:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.60@tcp1 to 192.168.204.60@tcp2 - Route table may be misconfigured [ 1551.284454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp2 192.168.204.60@tcp1 [ 1557.607698] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:44:24 (1787615064) [ 1558.000402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.037742] LNet: 77129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.047240] LNetError: 77129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.053307] LNet: Removed LNI 192.168.204.60@tcp1 [ 1559.080422] LNet: Removed LNI 192.168.204.60@tcp2 [ 1559.158870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.210686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.233670] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1560.152689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1560.173480] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1561.130810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1561.185730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.60@tcp [ 1561.220545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp1 [ 1561.257527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp -e network_timeout [ 1561.289614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -e network_timeout [ 1561.343692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -e network_timeout [ 1561.403438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -e network_timeout [ 1561.450304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1561.473901] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1561.489713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1561.575257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1561.676459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1562.272709] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1568.283991] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:44:34 (1787615074) [ 1568.638756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.669754] LNet: 77990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.680436] LNetError: 77990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.691386] LNetError: 77990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1568.704048] LNet: Removed LNI 192.168.204.60@tcp [ 1569.776343] LNet: Removed LNI 192.168.204.60@tcp1 [ 1569.847796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.975542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1570.372660] LNet: 78032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.798217] Key type .llcrypt unregistered [ 1570.806461] Key type ._llcrypt unregistered [ 1584.235415] Key type ._llcrypt registered [ 1584.248036] Key type .llcrypt registered [ 1586.773743] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:44:52 (1787615092) [ 1587.524274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.553920] alg: No test for adler32 (adler32-zlib) [ 1588.798628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.854757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1592.631229] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:44:59 (1787615099) [ 1593.117467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.150852] LNet: 79131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.179689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.250780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.281707] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1594.144232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1594.166810] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1595.331942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1595.389752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp1 [ 1595.486311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp1 [ 1602.081170] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:45:08 (1787615108) [ 1603.943726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg460-server [ 1605.613528] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:45:12 (1787615112) [ 1607.552479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg460-server [ 1609.306567] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:45:15 (1787615115) [ 1610.907412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg460-server [ 1612.582580] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:45:19 (1787615119) [ 1614.674581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg460-server [ 1616.588794] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:45:23 (1787615123) [ 1618.861511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg460-server [ 1620.566237] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:45:27 (1787615127) [ 1622.514318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg460-server [ 1624.838284] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:45:31 (1787615131) [ 1626.796385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg460-server [ 1628.624160] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:45:35 (1787615135) [ 1630.609315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg460-server [ 1632.903586] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:45:39 (1787615139) [ 1634.605650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg460-server [ 1636.730629] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:45:43 (1787615143) [ 1636.842994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.892259] LNet: 83002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.907884] LNetError: 83002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.920775] LNet: Removed LNI 192.168.204.60@tcp [ 1637.946138] LNetError: 83002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.018869] LNet: Removed LNI 192.168.204.60@tcp1 [ 1639.074512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.163026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.198673] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1640.032317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1640.151927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1641.627473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.662993] LNet: 83165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.667646] LNetError: 83165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.682744] LNet: Removed LNI 192.168.204.60@tcp [ 1641.707392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.792276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1641.838780] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1642.975427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1644.353242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.389487] LNet: 83319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.396095] LNetError: 83319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.413863] LNet: Removed LNI 192.168.204.60@tcp [ 1645.465487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.574874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.639808] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1646.699648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1646.820567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1647.105326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.171223] LNet: Removed LNI 192.168.204.60@tcp [ 1648.221409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.308412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1648.334312] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1649.372643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1649.660867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.688070] LNet: 83612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.697186] LNet: 83612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1649.706768] LNetError: 83612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.715586] LNetError: 83612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1649.736900] LNet: Removed LNI 192.168.204.60@tcp [ 1649.809386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.944890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.001057] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1651.111446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1651.198641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1651.450295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.522777] LNet: Removed LNI 192.168.204.60@tcp [ 1652.578712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.706239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1652.756760] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1653.996481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1655.537690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.614185] LNet: Removed LNI 192.168.204.60@tcp [ 1656.672788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.736513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.774861] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1657.604827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1657.713209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1658.031312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.077542] LNet: 84061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.089700] LNet: 84061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1658.095627] LNetError: 84061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.100660] LNetError: 84061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1659.124279] LNet: Removed LNI 192.168.204.60@tcp [ 1659.177058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.263948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1659.314795] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1660.211636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1660.541241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.608171] LNet: Removed LNI 192.168.204.60@tcp [ 1661.649429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.727673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.778605] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1662.620815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1662.731289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1663.037505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.103381] LNet: Removed LNI 192.168.204.60@tcp [ 1664.162178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.251290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1664.273935] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1664.947636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1665.219937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.278071] LNet: Removed LNI 192.168.204.60@tcp [ 1666.324547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.430783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.460991] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1667.507094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1667.635758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1669.232633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.331897] LNet: Removed LNI 192.168.204.60@tcp [ 1670.365421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.486878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1670.523844] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1671.384949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1672.928909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.024087] LNet: Removed LNI 192.168.204.60@tcp [ 1674.059042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.141857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.178674] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1674.722859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1674.796329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.831550] LNet: 84943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.841952] LNet: 84943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1674.849656] LNetError: 84943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.854933] LNetError: 84943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1674.870447] LNet: Removed LNI 192.168.204.60@tcp [ 1674.887924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.963513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.985485] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1675.714198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1675.790359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.835769] LNet: Removed LNI 192.168.204.60@tcp [ 1675.850940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.893258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1675.955715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.988242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.045413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1676.135371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.166855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.213788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1676.311568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.348387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.432521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1676.458531] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1677.176955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.220900] LNet: Removed LNI 192.168.204.60@tcp [ 1677.241724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.342513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1683.950518] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:46:30 (1787615190) [ 1684.084883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.162263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.211960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1684.233790] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1685.497052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.528938] LNet: Removed LNI 192.168.204.60@tcp231 [ 1685.543147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.601544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1685.634381] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1685.640263] LNet: 85978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1686.137530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1686.188283] LNet: Removed LNI 192.168.204.60@tcp231 [ 1686.214904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1686.277331] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1687.239990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.272642] LNet: Removed LNI 192.168.204.60@tcp231 [ 1687.295503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.340269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1687.364232] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1687.368184] LNet: 86131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.694433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1687.728093] LNet: Removed LNI 192.168.204.60@tcp231 [ 1687.748380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1687.777078] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1688.747448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.794590] LNet: Removed LNI 192.168.204.60@tcp231 [ 1688.817328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.879346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1688.912828] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1688.918739] LNet: 86282:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.317884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1689.367803] LNet: Removed LNI 192.168.204.60@tcp231 [ 1689.383401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1689.423443] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1690.448817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.501259] LNet: Removed LNI 192.168.204.60@tcp231 [ 1690.539555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.607971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1690.650618] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1691.072511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1691.106884] LNet: Removed LNI 192.168.204.60@tcp231 [ 1691.164377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1691.206629] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1699.301474] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:46:45 (1787615205) [ 1699.553334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.600334] LNet: Removed LNI 192.168.204.60@tcp231 [ 1699.650833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.744307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1699.773329] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1700.395131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1700.444074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1700.475282] LNet: Removed LNI 192.168.204.60@tcp232 [ 1700.493274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1700.545454] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1701.322798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1701.388877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1701.423641] LNet: Removed LNI 192.168.204.60@tcp232 [ 1701.439942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1701.477854] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1702.286959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1702.322787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1702.356171] LNet: Removed LNI 192.168.204.60@tcp232 [ 1702.383236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1702.418593] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1703.170605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1703.196653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1703.225333] LNet: Removed LNI 192.168.204.60@tcp232 [ 1703.249919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1709.238556] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:46:55 (1787615215) [ 1709.758943] LNet: 88271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.767453] LNet: 88271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1710.393617] Key type .llcrypt unregistered [ 1710.398610] Key type ._llcrypt unregistered [ 1715.502492] Key type ._llcrypt registered [ 1715.504202] Key type .llcrypt registered [ 1715.667140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1715.688959] alg: No test for adler32 (adler32-zlib) [ 1716.584788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1716.612928] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1724.439841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1731.868723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.915805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1731.941651] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1733.233971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1733.274236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1733.318394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1733.355769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1733.412849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1733.449129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1733.484764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1733.527196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1733.561758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1733.580653] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1733.589386] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1733.845569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1733.873717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1733.900749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1733.947265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1733.983427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1734.022305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1734.052815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1734.107769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1734.133311] LNet: 88937:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1734.165348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1734.216326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1734.263058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1734.334710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1734.424312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1734.458795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1734.863060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1734.888610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1734.912902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1734.942879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1734.973613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1735.000627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1735.026459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1735.054735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1735.090658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1735.106771] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1735.294102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1735.342589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1735.393530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1735.431072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1735.473845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1735.514274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1735.560224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1735.611407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1735.646273] LNet: 89057:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1735.679306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1735.748306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1735.812301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1735.859281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1735.939617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1736.002972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1736.416774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1736.440713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1736.468820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1736.507940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1736.571383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1736.622971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1736.678797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1736.743585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1736.798670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1736.839571] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1737.160375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1737.205728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1737.260331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1737.319530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1737.390905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1737.456422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1737.498525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1737.522667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1737.558197] LNet: 89177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1737.571150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1737.597339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1737.617076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1737.640934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1737.666536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1737.694444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1738.125968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1738.182302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1738.238335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1738.314918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1738.391679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1738.462605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1738.547392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1738.607934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1738.664315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1739.036073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1739.063252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1739.097037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1739.139660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1739.171142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1739.222395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1739.276545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1739.321666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1739.365236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1739.420610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1739.453940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1739.507739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1739.553319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1739.593349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1740.191560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1740.242707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1740.326210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1740.404831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1740.440877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1740.507334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1740.552386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1740.615390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1740.684676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1740.709963] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1740.719934] LNet: Skipped 1 previous similar message [ 1741.089375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1741.117649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1741.142697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1741.174420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1741.203383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1741.253853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1741.328879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1741.405567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1741.436525] LNet: 89418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1741.445940] LNet: 89418:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1741.468688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1741.511719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1741.549484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1741.582361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1741.613234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1741.657214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1742.192287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1742.252905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1742.281678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1742.357610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1742.431724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1742.468347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1742.506215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1742.540672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1742.597497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1742.633821] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1742.931701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1742.961553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1742.990378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.024967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1743.059502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1743.081476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1743.122919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1743.172396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1743.224461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1743.286423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1743.352720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1743.417760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1743.453761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1743.496177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1749.443277] LNet: 89616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.450743] LNetError: 89616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.458296] LNet: Removed LNI 192.168.204.60@tcp [ 1749.481632] LNet: Removed LNI 192.168.204.60@tcp1 [ 1750.174507] Key type .llcrypt unregistered [ 1750.177778] Key type ._llcrypt unregistered [ 1768.267724] Key type ._llcrypt registered [ 1768.270513] Key type .llcrypt registered [ 1769.282605] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:47:56 (1787615276) [ 1769.767482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.782748] alg: No test for adler32 (adler32-zlib) [ 1770.661160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.728848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.769182] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1771.464919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1771.501384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1771.516780] LNetError: 90594:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1771.571534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.60@tcp [ 1771.633348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1777.442691] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:48:03 (1787615283) [ 1777.827237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.882814] LNet: 91184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.898000] LNetError: 91184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.934292] LNet: Removed LNI 192.168.204.60@tcp [ 1777.964501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.051865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.081431] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1778.874291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1778.909701] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1779.967860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.60@tcp [ 1780.002651] LNetError: 91450:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1780.063606] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1780.067859] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1781.284566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.60@tcp [ 1788.078741] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:48:15 (1787615295) [ 1789.661130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg460-server [ 1791.214977] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:48:18 (1787615298) [ 1792.936238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg460-server [ 1794.642121] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:48:21 (1787615301) [ 1796.819430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg460-server [ 1798.914079] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:48:25 (1787615305) [ 1799.089661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.121090] LNet: 93066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.128677] LNetError: 93066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.138836] LNet: Removed LNI 192.168.204.60@tcp [ 1799.150290] LNet: Removed LNI 192.168.204.60@tcp1 [ 1799.185371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.270239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.306936] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1799.819384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1799.869731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1799.902537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1799.987705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1800.046947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1800.131150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1805.767817] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:48:32 (1787615312) [ 1805.906682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.936547] LNet: 93799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.945694] LNetError: 93799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.953240] LNetError: 93799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1805.966765] LNet: Removed LNI 192.168.204.60@tcp [ 1806.000119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.061887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.082562] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1812.298928] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:48:39 (1787615319) [ 1812.586451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.626804] LNet: 94508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.658733] LNetError: 94508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.677497] LNet: Removed LNI 192.168.204.60@tcp [ 1812.694916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.740296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.760718] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1819.411955] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:48:46 (1787615326) [ 1820.802875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1822.576591] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:48:49 (1787615329) [ 1822.969809] LNet: 95543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.973658] LNetError: 95543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.990247] LNet: Removed LNI 192.168.204.60@tcp [ 1823.630673] Key type .llcrypt unregistered [ 1823.634672] Key type ._llcrypt unregistered [ 1829.956640] Key type ._llcrypt registered [ 1829.959186] Key type .llcrypt registered [ 1830.140808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1830.159522] alg: No test for adler32 (adler32-zlib) [ 1831.085093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1831.113382] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1837.768325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1842.883349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.160@tcp [ 1843.402482] LNet: 95968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.411530] LNetError: 95968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.436208] LNet: Removed LNI 192.168.204.60@tcp [ 1843.897737] Key type .llcrypt unregistered [ 1843.900508] Key type ._llcrypt unregistered [ 1856.125050] Key type ._llcrypt registered [ 1856.138668] Key type .llcrypt registered [ 1857.550910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1858.962597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1860.435717] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:49:27 (1787615367) [ 1861.886611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1864.110398] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:49:30 (1787615370) [ 1865.935754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg460-server [ 1867.461751] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:49:34 (1787615374) [ 1869.037399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg460-server [ 1870.597780] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:49:37 (1787615377) [ 1871.849804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.952804] alg: No test for adler32 (adler32-zlib) [ 1873.007879] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1873.270671] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1874.983549] Key type lgssc registered [ 1876.560055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1895.542915] Key type lgssc unregistered [ 1895.751564] LNet: 99150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.759122] LNetError: 99150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.773745] LNet: Removed LNI 192.168.204.60@tcp [ 1896.428613] Key type .llcrypt unregistered [ 1896.431135] Key type ._llcrypt unregistered [ 1911.129617] Key type ._llcrypt registered [ 1911.132470] Key type .llcrypt registered [ 1912.467846] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:50:19 (1787615419) [ 1912.814539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1912.836426] alg: No test for adler32 (adler32-zlib) [ 1913.762338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.845188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1913.892657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1913.931728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1913.968852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1919.614499] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:50:26 (1787615426) [ 1919.888667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.904071] LNet: 100565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.931366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.993192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.019396] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1920.859409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.879672] LNet: 100710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.885678] LNetError: 100710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.902376] LNet: Removed LNI 192.168.204.60@tcp [ 1920.932159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.990502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1921.014614] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1921.459951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.485415] LNetError: 100738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.509179] LNet: Removed LNI 192.168.204.60@tcp [ 1921.526321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.602731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1921.635439] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1927.975742] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:50:34 (1787615434) [ 1928.275820] LNet: 101328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.281191] LNet: 101328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1928.296207] LNetError: 101328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.314984] LNet: Removed LNI 192.168.204.60@tcp [ 1928.794624] Key type .llcrypt unregistered [ 1928.803183] Key type ._llcrypt unregistered [ 1934.121205] Key type ._llcrypt registered [ 1934.125731] Key type .llcrypt registered [ 1934.267876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1934.286229] alg: No test for adler32 (adler32-zlib) [ 1935.225918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.282810] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1935.287635] LNet: *** cfs_fail_loc=e003, val=100*** [ 1935.958715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1935.982899] LNet: *** cfs_fail_loc=e003, val=100*** [ 1935.988931] LNet: Skipped 2 previous similar messages [ 1935.997780] Ping reply received with mapping: [ 1936.000032] found metadata reply NI[192.168.204.60@tcp] [ 1936.002225] 3c cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.005423] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.011459] 5a 5a 5a 5a 5a 5a 5a 5a [ 1936.223713] LNet: 101729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.231368] LNetError: 101729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.255069] LNet: Removed LNI 192.168.204.60@tcp [ 1937.911418] Key type .llcrypt unregistered [ 1937.914645] Key type ._llcrypt unregistered [ 1948.883509] Key type ._llcrypt registered [ 1948.886455] Key type .llcrypt registered [ 1950.173726] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:50:56 (1787615456) [ 1950.535393] Key type .llcrypt unregistered [ 1950.538592] Key type ._llcrypt unregistered [ 1956.536553] Key type ._llcrypt registered [ 1956.539371] Key type .llcrypt registered [ 1956.719898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1956.734361] alg: No test for adler32 (adler32-zlib) [ 1957.999549] LNetError: 102787:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1958.413681] Key type .llcrypt unregistered [ 1958.417328] Key type ._llcrypt unregistered [ 1963.619187] Key type ._llcrypt registered [ 1963.622770] Key type .llcrypt registered [ 1964.986800] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:51:11 (1787615471) [ 1965.584340] Key type .llcrypt unregistered [ 1965.586517] Key type ._llcrypt unregistered [ 1971.980408] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1971.984703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1974.492273] Key type ._llcrypt registered [ 1974.495370] Key type .llcrypt registered [ 1974.689664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1974.718967] alg: No test for adler32 (adler32-zlib) [ 1975.618951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.686518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1987.462580] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:51:33 (1787615493) [ 1987.614950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.648347] LNet: 104571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.662789] LNetError: 104571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1987.691614] LNet: Removed LNI 10.1.2.3@tcp [ 1987.719655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.726753] Key type .llcrypt unregistered [ 1988.729921] Key type ._llcrypt unregistered [ 1996.318220] Key type ._llcrypt registered [ 1996.320511] Key type .llcrypt registered [ 1996.665356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.695024] alg: No test for adler32 (adler32-zlib) [ 1997.598829] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1997.711912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1998.011318] LNet: 104869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.028222] LNetError: 104869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.064042] LNet: Removed LNI 192.168.204.60@tcp [ 1998.787649] Key type .llcrypt unregistered [ 1998.792190] Key type ._llcrypt unregistered [ 2011.759236] Key type ._llcrypt registered [ 2011.762278] Key type .llcrypt registered [ 2013.286939] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:51:59 (1787615519) [ 2014.206724] Key type .llcrypt unregistered [ 2014.215614] Key type ._llcrypt unregistered [ 2020.537074] Key type ._llcrypt registered [ 2020.541738] Key type .llcrypt registered [ 2020.748462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.768239] alg: No test for adler32 (adler32-zlib) [ 2029.254833] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:52:16 (1787615536) [ 2029.333298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.393703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.484207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.504615] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2030.389868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.459645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2030.501931] LNet: Removed LNI 192.168.204.60@tcp [ 2030.902874] LNet: 106638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.435374] Key type .llcrypt unregistered [ 2031.438841] Key type ._llcrypt unregistered [ 2046.892816] Key type ._llcrypt registered [ 2046.905614] Key type .llcrypt registered [ 2048.414730] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:52:34 (1787615554) [ 2048.847490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2048.874539] alg: No test for adler32 (adler32-zlib) [ 2049.860873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.916609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.942523] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2050.689275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2050.789131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.60@tcp [ 2050.832845] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2050.835382] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.841953] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2050, 0, 0 [ 2050.860937] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2050.865371] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2050.869663] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.881449] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2050, 0, 0 [ 2050.896970] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2050.902314] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2050.911412] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.913573] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2050, 0, 2100 [ 2050.927239] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2050.929063] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2050.937031] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2050.944479] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2050, 0, 2100 [ 2050.951446] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2057.647913] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:52:44 (1787615564) [ 2058.339783] LNet: 108178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.347700] LNetError: 108178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.372771] LNet: Removed LNI 192.168.204.60@tcp [ 2059.171610] Key type .llcrypt unregistered [ 2059.180828] Key type ._llcrypt unregistered [ 2065.771140] Key type ._llcrypt registered [ 2065.773268] Key type .llcrypt registered [ 2066.197600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2066.234674] alg: No test for adler32 (adler32-zlib) [ 2067.106406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2067.149246] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2074.400852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2082.366978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.397210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2082.415387] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 2083.377971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2083.580244] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2083.611086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.619855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2085.972416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2086.016246] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2098.751751] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:53:25 (1787615605) [ 2099.167642] LNet: 109693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.178594] LNetError: 109693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.255178] LNet: Removed LNI 192.168.204.60@tcp [ 2100.265026] LNetError: 109693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.288862] LNet: Removed LNI 192.168.204.60@tcp1 [ 2100.307414] LNet: Removed LNI 10.1.2.3@tcp99 [ 2101.258968] Key type .llcrypt unregistered [ 2101.262760] Key type ._llcrypt unregistered [ 2107.856620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2107.866265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2110.624583] Key type ._llcrypt registered [ 2110.629463] Key type .llcrypt registered [ 2110.992057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2111.024403] alg: No test for adler32 (adler32-zlib) [ 2112.054078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.157552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.210839] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2113.333530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2113.355722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2116.252493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2116.381574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2116.531876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2116.594489] LNetError: 110443:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2116.664727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2116.723687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2117.020881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2117.096661] LNet: 110463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.112606] LNetError: 110463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2117.118362] LNet: Removed LNI 10.1.2.3@tcp [ 2118.185309] LNet: Removed LNI 192.168.204.60@tcp [ 2119.116605] Key type .llcrypt unregistered [ 2119.119956] Key type ._llcrypt unregistered [ 2135.846065] Key type ._llcrypt registered [ 2135.850603] Key type .llcrypt registered [ 2137.403893] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:54:03 (1787615643) [ 2137.876956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.905899] alg: No test for adler32 (adler32-zlib) [ 2138.822418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.911921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.951832] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2139.983657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2140.087738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg460-client@tcp [ 2146.406830] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:54:13 (1787615653) [ 2146.634572] LNet: 112003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.640967] LNetError: 112003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.691392] LNet: Removed LNI 192.168.204.60@tcp [ 2148.425676] Key type .llcrypt unregistered [ 2148.427690] Key type ._llcrypt unregistered [ 2155.049663] Key type ._llcrypt registered [ 2155.052720] Key type .llcrypt registered [ 2155.285267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.320310] alg: No test for adler32 (adler32-zlib) [ 2156.266378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2156.293028] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2156.317884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2163.791826] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:54:30 (1787615670) [ 2165.624758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2167.893414] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:54:34 (1787615674) [ 2169.808042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2171.931266] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:54:38 (1787615678) [ 2172.340665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.375634] LNet: 113430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.383895] LNetError: 113430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.407755] LNet: Removed LNI 192.168.204.60@tcp [ 2172.435178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.539423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2172.569119] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2173.496344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2173.554974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2180.980241] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:54:47 (1787615687) [ 2181.384185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.426924] LNet: 114140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.441181] LNetError: 114140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.467518] LNet: Removed LNI 192.168.204.60@tcp [ 2181.510831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.586649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2181.639152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.483568] Key type .llcrypt unregistered [ 2182.492931] Key type ._llcrypt unregistered [ 2195.310617] Key type ._llcrypt registered [ 2195.313805] Key type .llcrypt registered [ 2196.882385] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:55:03 (1787615703) [ 2197.371240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2197.388288] alg: No test for adler32 (adler32-zlib) [ 2198.311907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.358296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.381428] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2199.053366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2199.102833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2205.845829] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:55:12 (1787615712) [ 2205.963312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.002051] LNet: 115694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.021293] LNetError: 115694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.044920] LNet: Removed LNI 192.168.204.60@tcp [ 2206.080182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.183479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2206.234313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2212.301684] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:55:18 (1787615718) [ 2212.404141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.437969] LNet: 116274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.471763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.552879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2212.890389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2212.913521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2212.965261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2213.486928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2213.540340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2213.603549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.619176] LNet: 116314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.632870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.675082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2214.013735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2214.039197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2214.111510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2214.424860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2214.448731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2219.417856] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:55:26 (1787615726) [ 2219.535705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.552799] LNet: 116911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.591542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.652903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.679545] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2220.268747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2220.290075] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2220.921996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2220.966977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2221.123389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.136895] LNetError: 117193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.140943] LNet: Removed LNI 192.168.204.60@tcp [ 2221.153294] LNet: Removed LNI 192.168.204.60@tcp2 [ 2221.182644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.229126] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2221.233560] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2221.238391] LNet: 117199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.633317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2221.790392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.817108] LNet: Removed LNI 192.168.204.60@tcp [ 2222.821889] LNetError: 117236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.828940] LNetError: 117236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2222.844935] LNet: Removed LNI 192.168.204.60@tcp2 [ 2222.870190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.922101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.954186] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2223.519073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2223.535884] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2224.274910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2224.325865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2224.379136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2224.616793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.638526] LNet: 117520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.646849] LNet: 117520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2225.695283] LNet: Removed LNI 192.168.204.60@tcp [ 2225.698228] LNetError: 117520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.710218] LNetError: 117520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2225.721939] LNet: Removed LNI 192.168.204.60@tcp2 [ 2225.741153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.820459] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2225.825957] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2225.831412] LNet: 117527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2226.338583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2226.543864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.560343] LNet: Removed LNI 192.168.204.60@tcp [ 2227.625372] LNet: Removed LNI 192.168.204.60@tcp2 [ 2227.663435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.711603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.733044] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2228.245089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2228.259891] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2228.980738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2229.025562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2229.244244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.303338] LNet: Removed LNI 192.168.204.60@tcp [ 2230.305633] LNetError: 117843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.321744] LNetError: 117843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2230.338588] LNet: Removed LNI 192.168.204.60@tcp2 [ 2230.353742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.413189] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2230.417656] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2230.420911] LNet: 117849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.772825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2230.930419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.967194] LNet: Removed LNI 192.168.204.60@tcp [ 2231.992609] LNet: Removed LNI 192.168.204.60@tcp2 [ 2232.014280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.072826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.091125] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2232.750409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2232.766731] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2233.535482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2233.570884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2233.625722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2233.820220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.836494] LNet: 118171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.841879] LNet: 118171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2234.911210] LNet: Removed LNI 192.168.204.60@tcp [ 2235.943428] LNet: Removed LNI 192.168.204.60@tcp2 [ 2235.984180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.059626] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2236.084101] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2236.088184] LNet: 118177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2236.572522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2236.809871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.837621] LNet: Removed LNI 192.168.204.60@tcp [ 2236.852841] LNet: Removed LNI 192.168.204.60@tcp2 [ 2236.876553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.922697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2236.949219] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2237.693191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2237.720733] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2238.403635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2238.437268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2238.549949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2238.860437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.890386] LNetError: 118510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.900046] LNetError: 118510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2238.907061] LNet: Removed LNI 192.168.204.60@tcp [ 2238.919694] LNet: Removed LNI 192.168.204.60@tcp2 [ 2238.963671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.047605] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2239.054541] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2239.553522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2239.991551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.055200] LNet: Removed LNI 192.168.204.60@tcp [ 2241.082689] LNet: Removed LNI 192.168.204.60@tcp2 [ 2241.124606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.176460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.207993] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2241.858487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2241.888637] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2242.665394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2242.703890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2242.726986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2242.780875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2243.115500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.191190] LNet: Removed LNI 192.168.204.60@tcp [ 2245.235481] LNet: Removed LNI 192.168.204.60@tcp2 [ 2245.277190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.360340] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2245.365278] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2245.376347] LNet: 118876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.388446] LNet: 118876:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2246.011651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2252.381318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2253.868123] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:56:00 (1787615760) [ 2253.971303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.001292] LNet: 119630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.012979] LNet: 119630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2254.024182] LNet: Removed LNI 192.168.204.60@tcp [ 2255.080399] LNet: Removed LNI 192.168.204.60@tcp2 [ 2255.107212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.381729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2255.400681] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2256.043354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2256.069751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2257.015440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.047803] LNetError: 119909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2257.052822] LNetError: 119909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2257.064416] LNet: Removed LNI 10.1.2.3@tcp [ 2257.088691] LNet: Removed LNI 192.168.204.60@tcp [ 2257.124825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.200929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2257.243620] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2257.256260] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2257.787816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2257.809893] LNet: Removed LNI 192.168.204.60@tcp [ 2257.823709] LNet: Removed LNI 10.1.2.3@tcp [ 2263.703824] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:56:10 (1787615770) [ 2263.903760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.989384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.053479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.074511] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2264.734707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2264.782684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2264.817990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2264.848850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2264.883877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2264.928868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2264.969671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2265.017293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2265.060391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2265.098957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2265.153305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2265.348665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2271.083873] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:56:17 (1787615777) [ 2271.279504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.368767] LNet: Removed LNI 192.168.204.60@tcp [ 2272.399894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.522737] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2272.627669] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2272.634300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2272.638283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2275.004510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.029249] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2275.789662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2275.831369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2277.899549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2277.932296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2277.999312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.60@tcp -m PUT -e local_timeout -r 1 [ 2278.064184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2278.144821] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2278.147367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2278.398859] LNetError: 121316:0:(lib-move.c:4801:lnet_parse()) 192.168.204.60@tcp, src 192.168.204.60@tcp: Dropping PUT (error -108 looking up sender) [ 2279.461491] LNet: Removed LNI 10.1.2.3@tcp [ 2280.489622] LNet: Removed LNI 192.168.204.60@tcp [ 2280.896466] Key type .llcrypt unregistered [ 2280.899506] Key type ._llcrypt unregistered [ 2292.559524] Key type ._llcrypt registered [ 2292.562373] Key type .llcrypt registered [ 2293.780541] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:56:40 (1787615800) [ 2294.202651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.230429] alg: No test for adler32 (adler32-zlib) [ 2295.176873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.556418] LNet: 122632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.236343] Key type .llcrypt unregistered [ 2296.240827] Key type ._llcrypt unregistered [ 2302.148100] Key type ._llcrypt registered [ 2302.155730] Key type .llcrypt registered [ 2302.377081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2302.396718] alg: No test for adler32 (adler32-zlib) [ 2303.382181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2303.444868] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2310.886706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2311.125437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2311.688261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 2311.994338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2312.040307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2312.087110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2312.132821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2312.167217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2312.202265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2312.240499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2312.266206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2312.284396] LNet: 123086:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2312.297946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2312.338814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2312.373658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2312.422929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2312.456882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2312.489563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2313.158773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2313.485345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2313.532365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2313.586939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2313.625353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2313.665180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2313.695473] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 2313.714701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2313.761852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2313.797361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2313.816553] LNet: 123203:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2313.832848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2313.862742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2313.896391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2313.931716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2313.964498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2313.996847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2314.764925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 2315.012356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.038438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.064167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.090822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.126674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2315.162389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2315.194517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2315.230968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2315.250481] LNet: 123320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.275144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2315.327761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2315.364281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2315.423960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2315.462862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2315.507875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2316.035597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2316.251334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2316.282411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2316.309142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2316.329611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2316.349188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2316.373410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2316.412185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2316.442187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2316.476847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2316.511146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2316.543274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2316.583936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2316.613369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2316.641838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2317.006714] LNet: 123487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.017633] LNetError: 123487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.035803] LNet: Removed LNI 192.168.204.60@tcp [ 2317.432816] Key type .llcrypt unregistered [ 2317.439103] Key type ._llcrypt unregistered [ 2333.396168] Key type ._llcrypt registered [ 2333.400369] Key type .llcrypt registered [ 2334.867941] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:57:21 (1787615841) [ 2335.344494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.376151] alg: No test for adler32 (adler32-zlib) [ 2336.343653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.719932] LNet: 124374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.351353] Key type .llcrypt unregistered [ 2337.359342] Key type ._llcrypt unregistered [ 2344.560670] Key type ._llcrypt registered [ 2344.563457] Key type .llcrypt registered [ 2344.812048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2344.829819] alg: No test for adler32 (adler32-zlib) [ 2345.804934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2345.836358] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2352.582527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2361.328503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.413353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2361.450248] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 2362.378876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2362.914739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 2363.189567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2363.231240] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 2363.249402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2363.285111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2363.313596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2363.355150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2363.387715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2363.429254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2363.455752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2363.470523] LNet: 125030:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.486245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2363.519914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2363.561504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2363.612971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2363.649882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2363.684908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2364.319613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2364.575208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2364.603910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2364.661991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.697388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.741769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.784700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.822621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.851203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.868727] LNet: 125147:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.886226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.906306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2364.963785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2365.004689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2365.040189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2365.095545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.776786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 2366.036803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2366.075893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2366.113170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2366.148485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2366.206512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2366.239676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2366.274803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2366.305662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2366.319562] LNet: 125264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.337059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2366.377564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2366.407896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2366.455109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2366.510343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2366.552506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2367.204120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2367.441850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2367.466927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2367.495114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2367.527633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2367.546826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2367.568434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2367.598356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2367.623873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2367.658486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2367.688360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2367.739332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2367.772846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2367.811964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2367.843857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2372.732447] LNet: 125466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.740387] LNetError: 125466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.747838] LNet: Removed LNI 192.168.204.60@tcp [ 2372.765228] LNet: Removed LNI 192.168.204.60@tcp1 [ 2373.206457] Key type .llcrypt unregistered [ 2373.210907] Key type ._llcrypt unregistered [ 2388.958434] Key type ._llcrypt registered [ 2388.961304] Key type .llcrypt registered [ 2390.111896] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:58:16 (1787615896) [ 2390.585784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2390.599362] alg: No test for adler32 (adler32-zlib) [ 2391.547554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.626824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.658796] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2392.356874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2392.382861] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2393.257130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2393.343176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2394.399191] LNet: Removed LNI 192.168.204.60@tcp [ 2400.928425] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:58:27 (1787615907) [ 2402.409275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2404.074896] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:58:30 (1787615910) [ 2410.992714] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:58:37 (1787615917) [ 2418.936832] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:58:45 (1787615925) [ 2427.704980] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:58:53 (1787615933) [ 2436.361570] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:59:02 (1787615942) [ 2444.486530] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:59:10 (1787615950) [ 2452.373949] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:59:18 (1787615958) [ 2461.404681] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:59:27 (1787615967) [ 2469.104727] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:59:35 (1787615975) [ 2476.781352] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:59:43 (1787615983) [ 2484.338784] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:59:50 (1787615990) [ 2491.857539] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:59:58 (1787615998) [ 2499.847281] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:00:06 (1787616006) [ 2506.505088] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:00:13 (1787616013) [ 2513.418344] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:00:20 (1787616020) [ 2520.014620] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:00:26 (1787616026) [ 2527.596196] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:00:34 (1787616034) [ 2534.406480] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:00:41 (1787616041) [ 2534.687518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2534.710754] LNet: 137160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.716627] LNetError: 137160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.785509] LNet: Removed LNI 192.168.204.60@tcp2 [ 2535.820363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.032559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2536.053034] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2537.112987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2537.132598] LNet: 137302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.137933] LNetError: 137302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.152790] LNet: Removed LNI 192.168.204.60@tcp [ 2537.176920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2537.233669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2537.250764] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2544.067985] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:00:50 (1787616050) [ 2551.517794] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:00:57 (1787616057) [ 2560.663368] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:01:07 (1787616067) [ 2568.530317] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:01:15 (1787616075) [ 2574.858425] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:01:21 (1787616081) [ 2581.994509] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:01:28 (1787616088) [ 2589.674143] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:01:36 (1787616096) [ 2597.063352] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:01:43 (1787616103) [ 2604.216348] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:01:51 (1787616111) [ 2612.291248] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:01:58 (1787616118) [ 2619.495639] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:02:06 (1787616126) [ 2626.303091] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:02:13 (1787616133) [ 2632.180232] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2352 sec ========== 20:02:18 (1787616138) [ 2632.486895] LNet: 144779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.498649] LNetError: 144779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.513225] LNet: Removed LNI 192.168.204.60@tcp [ 2633.047797] Key type .llcrypt unregistered [ 2633.050777] Key type ._llcrypt unregistered [ 2645.089773] Key type ._llcrypt registered [ 2645.110047] Key type .llcrypt registered [ 2645.371685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2645.389326] alg: No test for adler32 (adler32-zlib) [ 2646.483295] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2646.811816] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2648.519184] Key type lgssc registered [ 2649.916494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2709.842560] Lustre: Mounted lustre-client - version 2.17.57_80_g74d5152 [ 2714.694496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2721.138716] Lustre: Unmounted lustre-client [ 2741.066672] Key type lgssc unregistered [ 2741.338780] LNet: 147735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.348543] LNetError: 147735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.365614] LNet: Removed LNI 192.168.204.60@tcp [ 2742.025282] Key type .llcrypt unregistered [ 2742.030679] Key type ._llcrypt unregistered