[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516368133 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004026] kvm-guest: setup PV IPIs [ 0.007561] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.026177] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035077] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039103] *** VALIDATE tmpfs *** [ 0.042168] *** VALIDATE proc *** [ 0.043323] *** VALIDATE cgroup *** [ 0.044013] *** VALIDATE cgroup2 *** [ 0.045386] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.046215] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047035] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048054] Spectre V2 : User space: Vulnerable [ 0.049015] Speculative Store Bypass: Vulnerable [ 0.052635] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.055305] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056873] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057026] ... version: 2 [ 0.058012] ... bit width: 48 [ 0.059012] ... generic registers: 4 [ 0.060014] ... value mask: 0000ffffffffffff [ 0.061025] ... max period: 00007fffffffffff [ 0.062015] ... fixed-purpose events: 3 [ 0.063014] ... event mask: 000000070000000f [ 0.064324] rcu: Hierarchical SRCU implementation. [ 0.066895] smp: Bringing up secondary CPUs ... [ 0.067829] x86: Booting SMP configuration: [ 0.068024] .... node #0, CPUs: #1 #2 #3 [ 0.075641] smp: Brought up 1 node, 4 CPUs [ 0.077015] smpboot: Max logical packages: 1 [ 0.078013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.152311] node 0 deferred pages initialised in 69ms [ 0.155278] devtmpfs: initialized [ 0.158295] x86/mm: Memory block size: 128MB [ 0.169081] gcov: version magic: 0x41383552 [ 0.177005] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.187094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.195122] pinctrl core: initialized pinctrl subsystem [ 0.200000] [ 0.202013] ************************************************************* [ 0.209018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212014] ** ** [ 0.219022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224014] ** ** [ 0.228015] ** This means that this kernel is built to expose internal ** [ 0.233017] ** IOMMU data structures, which may compromise security on ** [ 0.239014] ** your system. ** [ 0.244021] ** ** [ 0.251016] ** If you see this message and you are not debugging the ** [ 0.255015] ** kernel, report this immediately to your vendor! ** [ 0.261014] ** ** [ 0.267013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272024] ************************************************************* [ 0.279046] NET: Registered protocol family 16 [ 0.284284] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.289073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302140] cpuidle: using governor menu [ 0.305169] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.308778] PCI: Using configuration type 1 for base access [ 0.312141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326072] cryptd: max_cpu_qlen set to 1000 [ 0.329147] ACPI: Added _OSI(Module Device) [ 0.331014] ACPI: Added _OSI(Processor Device) [ 0.333017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336260] ACPI: Added _OSI(Processor Aggregator Device) [ 0.345869] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.357729] ACPI: Interpreter enabled [ 0.359100] ACPI: PM: (supports S0 S3 S4 S5) [ 0.363053] ACPI: Using IOAPIC for interrupt routing [ 0.367150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372650] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.395013] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.399337] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.405346] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.413090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.423290] acpiphp: Slot [2] registered [ 0.425866] acpiphp: Slot [5] registered [ 0.428153] acpiphp: Slot [6] registered [ 0.430108] acpiphp: Slot [3] registered [ 0.432128] acpiphp: Slot [4] registered [ 0.433146] acpiphp: Slot [7] registered [ 0.434112] acpiphp: Slot [8] registered [ 0.436940] acpiphp: Slot [9] registered [ 0.439137] acpiphp: Slot [10] registered [ 0.441134] acpiphp: Slot [11] registered [ 0.444130] acpiphp: Slot [12] registered [ 0.446949] acpiphp: Slot [13] registered [ 0.449602] acpiphp: Slot [14] registered [ 0.454148] acpiphp: Slot [15] registered [ 0.457132] acpiphp: Slot [16] registered [ 0.458105] acpiphp: Slot [17] registered [ 0.461010] acpiphp: Slot [18] registered [ 0.464388] acpiphp: Slot [19] registered [ 0.467730] acpiphp: Slot [20] registered [ 0.470153] acpiphp: Slot [21] registered [ 0.473575] acpiphp: Slot [22] registered [ 0.476317] acpiphp: Slot [23] registered [ 0.480134] acpiphp: Slot [24] registered [ 0.482661] acpiphp: Slot [25] registered [ 0.486158] acpiphp: Slot [26] registered [ 0.487120] acpiphp: Slot [27] registered [ 0.489138] acpiphp: Slot [28] registered [ 0.492227] acpiphp: Slot [29] registered [ 0.495119] acpiphp: Slot [30] registered [ 0.498208] acpiphp: Slot [31] registered [ 0.501277] PCI host bridge to bus 0000:00 [ 0.504026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.506025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.509081] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.511024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.515026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.518026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.522589] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.528469] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.537000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.555015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.568071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.576023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.581019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.587021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.594000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.599222] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.606050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.610063] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.616095] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.623014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.642828] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.653830] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.669981] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.682021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.689028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.713016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.731783] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.741000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.749018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.770017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.784000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.793448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.802104] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.806485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.811725] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.819138] iommu: Default domain type: Passthrough [ 0.822221] SCSI subsystem initialized [ 0.825189] ACPI: bus type USB registered [ 0.828170] usbcore: registered new interface driver usbfs [ 0.831120] usbcore: registered new interface driver hub [ 0.834096] usbcore: registered new device driver usb [ 0.837158] pps_core: LinuxPPS API ver. 1 registered [ 0.839011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.843253] PTP clock support registered [ 0.846079] EDAC MC: Ver: 3.0.0 [ 0.847000] PCI: Using ACPI for IRQ routing [ 0.847991] NetLabel: Initializing [ 0.848013] NetLabel: domain hash size = 128 [ 0.849000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.852104] NetLabel: unlabeled traffic allowed by default [ 0.854451] vgaarb: loaded [ 0.856043] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.858010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.864375] clocksource: Switched to clocksource kvm-clock [ 1.158635] VFS: Disk quotas dquot_6.6.0 [ 1.160986] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.164320] *** VALIDATE ramfs *** [ 1.165738] *** VALIDATE hugetlbfs *** [ 1.167812] pnp: PnP ACPI init [ 1.170693] pnp: PnP ACPI: found 6 devices [ 1.188882] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.192635] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.195129] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.197387] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.200162] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.202790] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.205374] NET: Registered protocol family 2 [ 1.207903] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.212914] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.216361] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.221247] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.224121] TCP: Hash tables configured (established 65536 bind 65536) [ 1.226851] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.229118] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.231191] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.233745] NET: Registered protocol family 1 [ 1.235433] RPC: Registered named UNIX socket transport module. [ 1.237319] RPC: Registered udp transport module. [ 1.238695] RPC: Registered tcp transport module. [ 1.239787] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.242109] NET: Registered protocol family 44 [ 1.245215] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.247847] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.250389] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.253091] PCI: CLS 0 bytes, default 64 [ 1.255973] Unpacking initramfs... [ 3.475166] debug: unmapping init [mem 0xffff9dd23cc64000-0xffff9dd23ffcffff] [ 3.483541] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.489472] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.493408] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.147964] Initialise system trusted keyrings [ 4.149835] Key type blacklist registered [ 4.153192] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.223224] zbud: loaded [ 4.233285] *** VALIDATE nfs *** [ 4.236499] *** VALIDATE nfs4 *** [ 4.241155] pstore: using deflate compression [ 4.255287] Platform Keyring initialized [ 4.613580] NET: Registered protocol family 38 [ 4.618185] Key type asymmetric registered [ 4.621888] Asymmetric key parser 'x509' registered [ 4.628716] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.641510] io scheduler mq-deadline registered [ 4.645420] io scheduler kyber registered [ 4.648991] io scheduler bfq registered [ 4.654338] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.661614] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.669976] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.676915] ACPI: Power Button [PWRF] [ 4.694707] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.718201] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.752556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.788817] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.829477] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.839272] Non-volatile memory driver v1.3 [ 4.842676] Linux agpgart interface v0.103 [ 4.936625] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.943705] vda: detected capacity change from 0 to 74698752 [ 4.989799] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.995137] vdb: detected capacity change from 0 to 1073741824 [ 5.016512] libphy: Fixed MDIO Bus: probed [ 5.036466] usbcore: registered new interface driver usbserial_generic [ 5.049601] usbserial: USB Serial support registered for generic [ 5.059950] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.077414] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.086852] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.097104] mousedev: PS/2 mouse device common for all mice [ 5.106096] rtc_cmos 00:05: RTC can wake from S4 [ 5.118848] rtc_cmos 00:05: registered as rtc0 [ 5.124534] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.131743] intel_pstate: CPU model not supported [ 5.137472] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.144585] hid: raw HID events driver (C) Jiri Kosina [ 5.156758] usbcore: registered new interface driver usbhid [ 5.161519] usbhid: USB HID core driver [ 5.167185] drop_monitor: Initializing network drop monitor service [ 5.173796] Initializing XFRM netlink socket [ 5.181577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.184073] NET: Registered protocol family 10 [ 5.204608] Segment Routing with IPv6 [ 5.208644] NET: Registered protocol family 17 [ 5.217718] mpls_gso: MPLS GSO support [ 5.231463] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.246589] RAS: Correctable Errors collector initialized. [ 5.249362] AVX version of gcm_enc/dec engaged. [ 5.251028] AES CTR mode by8 optimization enabled [ 5.510952] sched_clock: Marking stable (5510850150, 0)->(6687583180, -1176733030) [ 5.516149] registered taskstats version 1 [ 5.518705] Loading compiled-in X.509 certificates [ 5.520873] zswap: loaded using pool lzo/zbud [ 5.590560] Key type big_key registered [ 5.633290] Key type encrypted registered [ 5.636718] ima: No TPM chip found, activating TPM-bypass! [ 5.641628] ima: Allocated hash algorithm: sha1 [ 5.646988] ima: No architecture policies found [ 5.649942] evm: Initialising EVM extended attributes: [ 5.653993] evm: security.selinux [ 5.656754] evm: security.ima [ 5.659707] evm: security.capability [ 5.661263] evm: HMAC attrs: 0x1 [ 5.664756] rtc_cmos 00:05: setting system clock to 2026-08-10 08:52:46 UTC (1786351966) [ 5.673340] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 5.675945] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 5.683220] Write protecting the kernel read-only data: 28672k [ 5.690507] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 5.697126] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 5.812647] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.827246] systemd[1]: Detected virtualization kvm. [ 5.828640] systemd[1]: Detected architecture x86-64. [ 5.830839] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.857641] systemd[1]: No hostname configured. [ 5.864536] systemd[1]: Set hostname to . [ 5.869733] random: systemd: uninitialized urandom read (16 bytes read) [ 5.876812] systemd[1]: Initializing machine ID from random generator. [ 6.131605] random: ln: uninitialized urandom read (6 bytes read) [ 6.643548] random: systemd: uninitialized urandom read (16 bytes read) [ 6.647821] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.662704] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.682896] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.618409] device-mapper: uevent: version 1.0.3 [ 8.620845] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 9.089363] hrtimer: interrupt took 2387996 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.521992] virtio_net virtio0 ens2: renamed from eth0 [ 11.750289] random: fast init done [ 11.924863] scsi host0: ata_piix [ 12.038642] scsi host1: ata_piix [ 12.039784] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 12.074479] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.218366] random: crng init done [ 17.219845] random: 7 urandom warning(s) missed due to ratelimiting [ 20.204114] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.207722] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.343316] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.276079] SELinux: Disabled at runtime. [ 28.364139] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.383213] systemd[1]: Detected virtualization kvm. [ 28.386625] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.163590] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.174924] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.196771] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.213950] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.218890] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.251063] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.277401] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ 30.484305] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 32.970479] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.234351] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.293369] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.431513] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.609284] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 43.827417] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 44.964746] NFS: Registering the id_resolver key type [ 44.971834] Key type id_resolver registered [ 44.979333] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg425-client login: [ 124.257940] libcfs: loading out-of-tree module taints kernel. [ 124.354504] Key type ._llcrypt registered [ 124.359609] Key type .llcrypt registered [ 124.637664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 124.648726] alg: No test for adler32 (adler32-zlib) [ 125.841753] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 126.388232] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 128.079188] Key type lgssc registered [ 129.601520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 239.123424] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166 [ 243.695518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.370614] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing check_logdir /tmp/testlogs/ [ 260.016726] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing yml_node [ 263.319182] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 264.672238] Lustre: lustre-OST0000-osc-ffff9dd284ee4800: disconnect after 23s idle [ 265.836749] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 268.004965] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 269.262836] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 10 04:57:08 EDT 2026 [ 281.998772] Lustre: Unmounted lustre-client [ 308.128929] Key type lgssc unregistered [ 308.321274] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.326645] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.340680] LNet: Removed LNI 192.168.204.25@tcp [ 308.908301] Key type .llcrypt unregistered [ 308.912232] Key type ._llcrypt unregistered [ 334.810812] Key type ._llcrypt registered [ 334.815534] Key type .llcrypt registered [ 335.098000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.129412] alg: No test for adler32 (adler32-zlib) [ 336.068367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 336.187143] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 336.235543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 340.935263] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 341.459580] LNet: 9297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.476784] LNetError: 9297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.505428] LNet: Removed LNI 192.168.204.25@tcp [ 342.081835] Key type .llcrypt unregistered [ 342.084309] Key type ._llcrypt unregistered [ 348.773679] Key type ._llcrypt registered [ 348.775983] Key type .llcrypt registered [ 349.997663] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:58:29 (1786352309) [ 351.047466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.061903] alg: No test for adler32 (adler32-zlib) [ 351.954790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.031867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.060325] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.084623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.140385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.158111] LNet: 9719:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 358.535678] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:58:37 (1786352317) [ 358.808672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.840711] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.869555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.943726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 366.600722] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:58:46 (1786352326) [ 367.002324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.028137] LNet: 10882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.050899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.117445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 375.114385] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:58:54 (1786352334) [ 375.227310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.279724] LNet: 11460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.317577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.440322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 382.707942] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:59:02 (1786352342) [ 383.088103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.122129] LNet: 12054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.158569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.255459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 383.779956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 391.197608] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:59:10 (1786352350) [ 391.313864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.344102] LNet: 12636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.383103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.532456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 392.095556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 392.431247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 399.088894] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:59:18 (1786352358) [ 399.229995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.319234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.473243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 406.854373] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:59:26 (1786352366) [ 407.059444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.137406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.212585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 407.278759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.603837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 407.647639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.001755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 408.042276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.376868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 408.407724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.691473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 408.729720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 409.104455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 409.186624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.642225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 409.670826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.918816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 409.949849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 410.236739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 410.270589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 410.529214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 410.563904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.879433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.926981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 411.367613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 411.474279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.849639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.890616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.776410] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:59:38 (1786352378) [ 420.051845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.127959] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.130277] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 420.177420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.289139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 420.334145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 427.684114] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:59:47 (1786352387) [ 427.887535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.930971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.998604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 428.036847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 434.915597] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:59:54 (1786352394) [ 435.335174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.379477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.490940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 435.540842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 443.441764] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 05:00:02 (1786352402) [ 443.641257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.871576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.103159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.165651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 452.336162] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 05:00:11 (1786352411) [ 452.692863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.734199] LNet: 16853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.741487] LNet: 16853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 452.784329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.929501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 452.986118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 460.365256] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 05:00:20 (1786352420) [ 460.717543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.749385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.836881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.888556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.012110] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 05:00:27 (1786352427) [ 468.119256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.156367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.250483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.757452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.106289] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 05:00:36 (1786352436) [ 477.282068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.362190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.674320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.961203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.274777] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 05:00:44 (1786352444) [ 485.507371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.604098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.854969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.079326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.337726] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 05:00:52 (1786352452) [ 493.436300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.503439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.987306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.186413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.093480] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 05:01:01 (1786352461) [ 502.224832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.285465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.410454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.480628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.120451] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 05:01:09 (1786352469) [ 510.254801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.326517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.461604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.509978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.070960] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 05:01:17 (1786352477) [ 518.164600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.205639] LNet: 21716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.211780] LNet: 21716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 518.255575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.343242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 525.554473] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 05:01:24 (1786352484) [ 525.944361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.004849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.065044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 533.349525] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 05:01:33 (1786352493) [ 533.521018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.565750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.646880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 533.962153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 540.804247] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 05:01:39 (1786352499) [ 540.918025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.973759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.084336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 541.136308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 549.181928] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 05:01:48 (1786352508) [ 549.348027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.419893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.546519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 549.579956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 556.424668] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 05:01:55 (1786352515) [ 556.763559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.816539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.924078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 556.974196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 564.186976] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 05:02:03 (1786352523) [ 564.302388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.366628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.454697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.491280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.251133] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 05:02:10 (1786352530) [ 571.340754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.402049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.471059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 571.538606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 571.563444] LNetError: 25839:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 571.616972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 571.652473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 577.940769] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 05:02:17 (1786352537) [ 578.130691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.193442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.287097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wHTkpnhRhe [ 578.371101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wHTkpnhRhe [ 578.408357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wHTkpnhRhe [ 584.622378] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 05:02:24 (1786352544) [ 584.710534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.757179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.804484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 584.840072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 584.887866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 591.121595] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 05:02:30 (1786352550) [ 591.406376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.433961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.495246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.534105] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 593.104933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 594.458765] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 596.527210] Key type lgssc registered [ 597.844461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.205463] LNetError: 27600:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.25@tcp99 has no available nets [ 603.551311] Key type lgssc unregistered [ 603.904715] LNetError: 28707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.969991] LNet: Removed LNI 192.168.204.25@tcp [ 605.561739] Key type .llcrypt unregistered [ 605.563543] Key type ._llcrypt unregistered [ 612.199932] Key type ._llcrypt registered [ 612.201790] Key type .llcrypt registered [ 613.324885] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 05:02:52 (1786352572) [ 613.847862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.883660] alg: No test for adler32 (adler32-zlib) [ 614.814825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.883593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 616.986598] LNet: Message forwarding will be enabled in 2s [ 620.063166] LNet: Message forwarding enabled [ 620.551275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.569821] LNet: Message forwarding will stop in 2s to 6s [ 625.133905] LNet: Message forwarding disabled [ 625.239611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.770772] LNet: Message forwarding will be enabled in 2s [ 630.240078] LNet: Message forwarding enabled [ 634.585208] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 05:03:14 (1786352594) [ 634.759465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.793897] LNet: 30115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.796696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.918404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 643.015933] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 05:03:22 (1786352602) [ 643.169291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.193727] LNet: 30690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.219606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.280731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 643.346579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 643.375428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 643.401523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 643.424716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 643.447685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 643.468675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 643.498746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 643.547879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 643.600032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 643.665368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 643.699982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 643.746121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 643.789728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 643.820561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 643.869514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 643.914251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 643.963480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 644.008444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 644.032798] LNetError: 30740:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 650.404667] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 05:03:29 (1786352609) [ 650.504810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.533596] LNet: 31304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.586781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.700767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.170289] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 05:03:36 (1786352616) [ 659.437381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg425-server [ 661.092365] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 05:03:40 (1786352620) [ 661.439068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.477937] LNet: 32242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.499486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.573664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.614710] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 662.491991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[26-30]@tcp [ 662.531125] LNet: 32375:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 662.587544] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 662.615350] LNetError: 32246:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.29@tcp has gone from up to down [ 663.598369] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.27@tcp is being used as a gateway but routing feature is not turned on [ 663.613711] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 664.611500] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.27@tcp is being used as a gateway but routing feature is not turned on [ 664.634560] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 665.698659] LNetError: 32250:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.30@tcp has gone from up to down [ 665.718278] LNetError: 32250:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 666.686689] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.27@tcp is being used as a gateway but routing feature is not turned on [ 666.725699] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 670.243129] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 05:03:49 (1786352629) [ 670.839434] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 670.851182] LNetError: 32246:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 672.025976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg425-server [ 674.156896] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 05:03:53 (1786352633) [ 674.238582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.270237] LNet: 33304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.280842] LNetError: 33304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.305187] LNet: Removed LNI 192.168.204.25@tcp [ 674.331109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.406581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.424255] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 675.039412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[26-31/2]@tcp [ 675.057843] LNet: 33428:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 675.064446] LNet: 33428:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 675.103114] LNetError: 33315:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.28@tcp has gone from up to down [ 675.125670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[26-31/2]@tcp [ 680.989519] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 05:04:00 (1786352640) [ 681.205372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.227281] LNetError: 34021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.242237] LNet: Removed LNI 192.168.204.25@tcp [ 681.258609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.359966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 681.414925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.442712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.488893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 681.525439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.555161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.600526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 681.669297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.700856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.756960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 681.820763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.858941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.898173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 681.966797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.996419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.045117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.107818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.138470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.188716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 682.260343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.279190] LNet: 34114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.288393] LNet: 34114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 682.308905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.379145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.446968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.475303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.518984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 682.579736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.608878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.660656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.726794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.766766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.819653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 682.882296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.924356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.984648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.588649] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 05:04:08 (1786352648) [ 690.384994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg425-server [ 691.986335] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 05:04:11 (1786352651) [ 694.203412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg425-server [ 695.764725] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 05:04:15 (1786352655) [ 696.461122] Key type .llcrypt unregistered [ 696.462920] Key type ._llcrypt unregistered [ 703.153899] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 703.381515] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 703.384091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 703.390310] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 706.000364] Key type ._llcrypt registered [ 706.002547] Key type .llcrypt registered [ 706.182876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.198917] alg: No test for adler32 (adler32-zlib) [ 707.105959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.176809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.199190] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 707.778178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.795450] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.706878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 709.727807] LNet: Removed LNI 10.1.2.3@tcp [ 709.895379] LNet: 36144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.910707] LNetError: 36144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.936745] LNet: Removed LNI 192.168.204.25@tcp [ 710.560273] Key type .llcrypt unregistered [ 710.562668] Key type ._llcrypt unregistered [ 721.841656] Key type ._llcrypt registered [ 721.843728] Key type .llcrypt registered [ 723.036454] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 05:04:42 (1786352682) [ 723.532355] Key type .llcrypt unregistered [ 723.534386] Key type ._llcrypt unregistered [ 729.035189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 729.203695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 729.206138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 729.208974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.696838] Key type ._llcrypt registered [ 731.699885] Key type .llcrypt registered [ 731.847421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.857657] alg: No test for adler32 (adler32-zlib) [ 732.737686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.788961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.812457] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 733.405739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 733.431687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 736.292912] LNet: 37646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.299415] LNetError: 37646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 736.309127] LNet: Removed LNI 10.1.2.3@tcp [ 736.319818] LNet: Removed LNI 192.168.204.25@tcp [ 736.979490] Key type .llcrypt unregistered [ 736.983684] Key type ._llcrypt unregistered [ 749.040802] Key type ._llcrypt registered [ 749.043468] Key type .llcrypt registered [ 750.372553] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 05:05:09 (1786352709) [ 751.085123] Key type .llcrypt unregistered [ 751.090524] Key type ._llcrypt unregistered [ 755.879104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.099213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.103630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.110603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.616234] Key type ._llcrypt registered [ 757.625287] Key type .llcrypt registered [ 757.809402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.833542] alg: No test for adler32 (adler32-zlib) [ 758.797633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.888264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 758.925934] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 760.952090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.978079] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.881231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 762.898373] LNet: Removed LNI 10.1.2.3@tcp [ 762.910062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 762.929745] LNet: Removed LNI 10.1.2.31@tcp [ 762.942714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.960718] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.623445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 763.639369] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 764.555246] LNet: 39583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.569633] LNetError: 39583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 764.579507] LNet: Removed LNI 10.1.2.31@tcp [ 764.589215] LNet: Removed LNI 10.1.2.3@tcp [ 765.256634] Key type .llcrypt unregistered [ 765.267203] Key type ._llcrypt unregistered [ 777.493435] Key type ._llcrypt registered [ 777.495525] Key type .llcrypt registered [ 778.947789] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 05:05:38 (1786352738) [ 779.519709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.540324] alg: No test for adler32 (adler32-zlib) [ 780.465843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.541574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.567213] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 781.281468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.301323] LNet: 40542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.315358] LNetError: 40542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.341076] LNet: Removed LNI 192.168.204.25@tcp [ 781.360737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.438422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.25@tcp [ 781.480889] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 782.338435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.25@tcp [ 782.383675] LNet: Removed LNI 192.168.204.25@tcp [ 782.456908] LNet: 40677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.101598] Key type .llcrypt unregistered [ 783.104906] Key type ._llcrypt unregistered [ 793.614568] Key type ._llcrypt registered [ 793.616166] Key type .llcrypt registered [ 795.065648] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 05:05:54 (1786352754) [ 797.052973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg425-server [ 799.164800] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 05:05:58 (1786352758) [ 799.596342] Key type .llcrypt unregistered [ 799.602185] Key type ._llcrypt unregistered [ 804.774769] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.883749] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.890608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 804.896486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.335695] Key type ._llcrypt registered [ 807.339663] Key type .llcrypt registered [ 807.475387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.491789] alg: No test for adler32 (adler32-zlib) [ 808.387652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.431520] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 809.044284] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 811.119787] LNet: 42520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.127794] LNetError: 42520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.134622] LNet: Removed LNI 192.168.204.25@tcp1 [ 811.148399] LNet: Removed LNI 10.1.2.3@tcp2 [ 811.650565] Key type .llcrypt unregistered [ 811.652458] Key type ._llcrypt unregistered [ 821.394705] Key type ._llcrypt registered [ 821.397159] Key type .llcrypt registered [ 822.547804] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 05:06:22 (1786352782) [ 822.913311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.932467] alg: No test for adler32 (adler32-zlib) [ 823.818428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.863386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.880446] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 824.639225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.667862] LNet: 43494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.674502] LNetError: 43494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.691326] LNet: Removed LNI 192.168.204.25@tcp [ 824.717979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.801326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 824.826118] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 830.927903] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 05:06:30 (1786352790) [ 831.086836] LNet: 44076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.096600] LNetError: 44076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.125938] LNet: Removed LNI 192.168.204.25@tcp [ 831.705858] Key type .llcrypt unregistered [ 831.709864] Key type ._llcrypt unregistered [ 835.804573] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.873184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.878262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.885236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 838.370980] Key type ._llcrypt registered [ 838.373332] Key type .llcrypt registered [ 838.507383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.519823] alg: No test for adler32 (adler32-zlib) [ 839.413503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.461408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.478452] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 840.182952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.205458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.577816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.607629] LNet: 44832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.613909] LNetError: 44832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.620381] LNet: Removed LNI 10.1.2.3@tcp [ 842.634852] LNet: Removed LNI 192.168.204.25@tcp [ 842.659745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.727334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 842.750902] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 842.755043] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.136820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.153331] LNet: 44859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.157982] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.163612] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 843.168989] LNet: Removed LNI 10.1.2.3@tcp [ 843.179438] LNet: Removed LNI 192.168.204.25@tcp [ 843.204793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.282886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 843.321110] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 843.325781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.777632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 843.799930] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 844.714306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.730551] LNet: 45022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.736292] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 844.740874] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 844.745881] LNet: Removed LNI 10.1.2.3@tcp [ 844.748556] LNet: Removed LNI 192.168.204.25@tcp [ 844.758417] LNet: Removed LNI 192.168.204.25@tcp2 [ 844.780458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.838988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 844.862593] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 844.866292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.874944] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 851.082812] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 05:06:50 (1786352810) [ 851.163583] LNet: 45610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.170623] LNetError: 45610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.178949] LNetError: 45610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 851.184516] LNet: Removed LNI 10.1.2.3@tcp [ 851.186609] LNet: Removed LNI 192.168.204.25@tcp [ 851.213407] LNet: Removed LNI 192.168.204.25@tcp2 [ 851.908801] Key type .llcrypt unregistered [ 851.911823] Key type ._llcrypt unregistered [ 856.177865] Key type ._llcrypt registered [ 856.180893] Key type .llcrypt registered [ 856.338825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.356323] alg: No test for adler32 (adler32-zlib) [ 857.242534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.301192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.321124] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 858.042759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.065573] LNet: 46016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.075285] LNetError: 46016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.092401] LNet: Removed LNI 192.168.204.25@tcp [ 858.109721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.174621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 858.200124] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 858.561901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.580193] LNet: 46046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.590352] LNetError: 46046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.608832] LNet: Removed LNI 192.168.204.25@tcp [ 858.633130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.698691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 858.727863] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 859.144749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.171697] LNet: Removed LNI 192.168.204.25@tcp [ 859.191266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.246943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.271929] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 859.863623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.032103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.034641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.041313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.395257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.412366] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.417520] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 862.421133] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.425096] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.440232] LNet: Removed LNI 192.168.204.25@tcp [ 862.465698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.547382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 862.581572] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 869.133416] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 05:07:08 (1786352828) [ 869.260128] LNet: 46745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.273136] LNetError: 46745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.297626] LNet: Removed LNI 192.168.204.25@tcp [ 870.146338] Key type .llcrypt unregistered [ 870.151388] Key type ._llcrypt unregistered [ 875.151134] Key type ._llcrypt registered [ 875.153454] Key type .llcrypt registered [ 875.301161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.311653] alg: No test for adler32 (adler32-zlib) [ 876.175913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.212970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.230836] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 876.911826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.933729] LNet: 47157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.936648] LNetError: 47157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.947421] LNet: Removed LNI 192.168.204.25@tcp [ 876.965719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.013759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 877.035886] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 882.530258] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 05:07:22 (1786352842) [ 882.604113] LNet: 47740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.619710] LNetError: 47740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.641979] LNet: Removed LNI 192.168.204.25@tcp [ 883.392769] Key type .llcrypt unregistered [ 883.400154] Key type ._llcrypt unregistered [ 888.281653] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 888.327523] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 888.331750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 888.335239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 891.053445] Key type ._llcrypt registered [ 891.056552] Key type .llcrypt registered [ 891.234466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.257736] alg: No test for adler32 (adler32-zlib) [ 892.175968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.231703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.252370] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 893.074409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.093074] LNet: 48202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.097864] LNetError: 48202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.112644] LNet: Removed LNI 192.168.204.25@tcp [ 893.132587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.203513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 893.226071] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 899.702594] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 05:07:39 (1786352859) [ 899.798409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.818881] LNet: 48795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.823268] LNetError: 48795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.839447] LNet: Removed LNI 192.168.204.25@tcp [ 899.875773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.977464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 905.375237] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 05:07:44 (1786352864) [ 905.495254] LNet: 49377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.375110] Key type .llcrypt unregistered [ 906.376821] Key type ._llcrypt unregistered [ 911.447907] Key type ._llcrypt registered [ 911.449784] Key type .llcrypt registered [ 911.681708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.717225] alg: No test for adler32 (adler32-zlib) [ 912.643253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.701280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.725643] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 913.478846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.498580] LNet: 49784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.503660] LNetError: 49784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.519703] LNet: Removed LNI 192.168.204.25@tcp [ 913.550495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.600281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 913.624637] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 918.834581] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 05:07:58 (1786352878) [ 918.887987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.903175] LNet: 50371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.913581] LNetError: 50371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.930185] LNet: Removed LNI 192.168.204.25@tcp [ 918.953183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.025806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 924.372988] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 05:08:03 (1786352883) [ 925.780890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 927.532897] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 05:08:06 (1786352886) [ 929.016669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 930.936774] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 05:08:10 (1786352890) [ 932.354744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 934.083394] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 05:08:13 (1786352893) [ 935.556483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 937.135252] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 05:08:16 (1786352896) [ 938.397249] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 940.142577] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 05:08:19 (1786352899) [ 940.252326] LNet: 52419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.954874] Key type .llcrypt unregistered [ 940.959713] Key type ._llcrypt unregistered [ 945.761438] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.940348] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.959862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.966418] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.423329] Key type ._llcrypt registered [ 948.425314] Key type .llcrypt registered [ 948.567553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.583629] alg: No test for adler32 (adler32-zlib) [ 949.486667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.561609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.583778] LNet: Added LNI 192.168.204.25@tcp [8/512/0/180] [ 949.590319] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 949.593480] LNet: 52751:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.689530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.708510] LNet: 52770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.713848] LNetError: 52770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.718783] LNet: Removed LNI 10.1.2.3@tcp [ 949.721844] LNetError: 52770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.733704] LNet: Removed LNI 192.168.204.25@tcp [ 949.748945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.799699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.822746] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 949.883651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.909350] LNet: 52795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.913356] LNetError: 52795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.926923] LNet: Removed LNI 192.168.204.25@tcp [ 949.947584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.005085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.023755] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 950.029945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.033760] LNet: 52803:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.123357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.142821] LNet: 52821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.149543] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.155829] LNet: Removed LNI 10.1.2.3@tcp [ 950.158830] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.175517] LNet: Removed LNI 192.168.204.25@tcp [ 950.202671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.266400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.284933] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 950.290388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.292440] LNet: 52829:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.447605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.464351] LNet: 52855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.469760] LNetError: 52855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.477090] LNet: Removed LNI 10.1.2.3@tcp [ 950.479595] LNetError: 52855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.494545] LNet: Removed LNI 192.168.204.25@tcp [ 950.524897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.588303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.609383] LNet: Added LNI 192.168.204.25@tcp [8/256/0/182] [ 950.614559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 950.701738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.736196] LNet: 52883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.743270] LNetError: 52883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.748170] LNet: Removed LNI 10.1.2.3@tcp [ 950.750340] LNetError: 52883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.762588] LNet: Removed LNI 192.168.204.25@tcp [ 950.790473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.863644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.884469] LNet: Added LNI 192.168.204.25@tcp [8/256/0/182] [ 950.891365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 950.893967] LNet: 52891:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.980808] LNet: Removed LNI 10.1.2.3@tcp [ 950.993979] LNet: Removed LNI 192.168.204.25@tcp [ 951.548169] Key type .llcrypt unregistered [ 951.551786] Key type ._llcrypt unregistered [ 962.067828] Key type ._llcrypt registered [ 962.071157] Key type .llcrypt registered [ 963.253481] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 05:08:42 (1786352922) [ 964.794323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 966.629780] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 05:08:45 (1786352925) [ 967.082647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.098367] alg: No test for adler32 (adler32-zlib) [ 968.029975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.072568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.095759] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 968.899427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.25 [ 968.947374] LNet: 54158:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 968.958151] LNet: Removed LNI 192.168.204.25@tcp [ 975.166818] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 05:08:54 (1786352934) [ 975.298885] LNet: 54722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.305686] LNetError: 54722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.322309] LNet: Removed LNI 192.168.204.25@tcp [ 976.057475] Key type .llcrypt unregistered [ 976.059185] Key type ._llcrypt unregistered [ 980.560912] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.703378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.297624] Key type ._llcrypt registered [ 983.301221] Key type .llcrypt registered [ 983.478644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.489537] alg: No test for adler32 (adler32-zlib) [ 984.650721] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.853160] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 05:09:09 (1786352949) [ 989.993583] LNet: 55636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.998060] LNetError: 55636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.017882] LNet: Removed LNI 10.1.2.3@tcp [ 990.646787] Key type .llcrypt unregistered [ 990.648411] Key type ._llcrypt unregistered [ 994.715490] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.979407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.982582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.986565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 997.541289] Key type ._llcrypt registered [ 997.543744] Key type .llcrypt registered [ 997.709217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.724895] alg: No test for adler32 (adler32-zlib) [ 998.626871] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.409930] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 05:09:23 (1786352963) [ 1004.695349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1006.087265] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:09:25 (1786352965) [ 1006.176739] LNet: 56824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.181795] LNetError: 56824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.201096] LNet: Removed LNI 10.1.2.3@tcp [ 1006.745273] Key type .llcrypt unregistered [ 1006.749732] Key type ._llcrypt unregistered [ 1010.636741] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.813597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.816654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.820485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1013.404847] Key type ._llcrypt registered [ 1013.407840] Key type .llcrypt registered [ 1013.563726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.578479] alg: No test for adler32 (adler32-zlib) [ 1014.442722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.493769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.597701] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:09:40 (1786352980) [ 1020.937340] LNet: 58064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.943907] LNetError: 58064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.960640] LNet: Removed LNI 10.1.2.3@tcp [ 1021.480271] Key type .llcrypt unregistered [ 1021.484358] Key type ._llcrypt unregistered [ 1025.223520] Key type ._llcrypt registered [ 1025.225401] Key type .llcrypt registered [ 1025.351737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.364194] alg: No test for adler32 (adler32-zlib) [ 1026.248188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1026.282216] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1031.272978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1031.680597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1031.702262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1031.722616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1031.738785] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1031.854975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.875477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1031.892835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.912324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.943489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.962593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.989060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.007182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.022957] LNet: 58505:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.033439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.055679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.079490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.100545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.124195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.144316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.168673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.498891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1032.522481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1032.546663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1032.559791] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1032.687590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.707390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.731191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.751901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.772446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.787191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.807128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.828904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.843745] LNet: 58615:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.859754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.886939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.906351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.931762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.958601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.981410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.999843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.284868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1033.310498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1033.338153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1033.439532] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1033.476725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.494401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1033.516651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.536673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.555183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.577149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.599377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.620837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.639817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.658495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.675521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.698290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.724478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.746557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.769624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.075649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1034.110946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1034.136681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1034.151866] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1034.156615] LNet: Skipped 2 previous similar messages [ 1034.300506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.329840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.356127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.382877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.406691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.433135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.459676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.493811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.510531] LNet: 58833:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.517169] LNet: 58833:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1034.530847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.554742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.580965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.609029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.638633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.704763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.733682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.061311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1035.091602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1035.125245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1035.295285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.320345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1035.337724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.359974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.380804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.401606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.424307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.449916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.477950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.503470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.526387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.553874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.585967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.613550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.640626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1036.049984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1036.082551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1036.104889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1036.262496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.292136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1036.319083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.348648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.369298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.394085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.421278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.445870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.470572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.498873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.523737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.545168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.575854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.608378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.635850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.851543] LNet: 59096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.858822] LNetError: 59096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.925725] LNet: Removed LNI 192.168.204.25@tcp [ 1038.195617] Key type .llcrypt unregistered [ 1038.198932] Key type ._llcrypt unregistered [ 1048.056752] Key type ._llcrypt registered [ 1048.059305] Key type .llcrypt registered [ 1048.990819] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:10:08 (1786353008) [ 1049.549404] Key type .llcrypt unregistered [ 1049.551485] Key type ._llcrypt unregistered [ 1053.241361] Key type ._llcrypt registered [ 1053.243753] Key type .llcrypt registered [ 1053.426584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.443859] alg: No test for adler32 (adler32-zlib) [ 1054.310784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.334501] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1059.070284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1065.252882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.288493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1065.301389] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1066.415250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1066.448555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 1 [ 1066.497780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1066.538258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 1 [ 1066.582898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1066.625842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 1 [ 1066.659579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1066.681730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 1 [ 1066.706966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1066.720199] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1066.724841] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1066.958093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.987842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.015576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.041940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.069697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.097087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.123570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.149574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.164982] LNet: 60568:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.178642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.204140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.229483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.255600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.279024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.310830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.342575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1067.855982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1067.891441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 1 [ 1067.926177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1067.968670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 1 [ 1067.997352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1068.032664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 1 [ 1068.068889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1068.099420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 1 [ 1068.152973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1068.195704] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1068.206850] LNet: Skipped 2 previous similar messages [ 1068.481662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.502826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.529172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.555109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.597350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.627154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.654467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.686753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.707990] LNet: 60690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.724121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.749529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.769705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.796455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.831494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.856814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.900774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1069.346513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1069.377627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 1 [ 1069.406752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1069.454554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 1 [ 1069.493675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1069.516098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 1 [ 1069.541246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1069.567380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 1 [ 1069.593285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1069.609194] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1069.613674] LNet: Skipped 2 previous similar messages [ 1069.756523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.781479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.803743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.826635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.853743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.880526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.902677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.928825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.943381] LNet: 60811:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.963653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.984502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.011449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.037554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.066875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.092916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.122663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1070.464972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1070.487283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 1 [ 1070.514978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1070.543862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 1 [ 1070.570861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1070.605857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 1 [ 1070.647575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1070.686159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 1 [ 1070.712921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1070.840221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.857510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1070.878982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.900274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.925935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.944740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.962948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.981431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.005146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.024714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.060706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.091860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.112646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.136257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.158063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1071.525775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1071.546834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 1 [ 1071.573121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1071.597993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 1 [ 1071.629413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1071.658711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 1 [ 1071.687191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1071.717224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 1 [ 1071.753195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1071.776964] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1071.782073] LNet: Skipped 5 previous similar messages [ 1071.979285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.005370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.026899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.053909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.076959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.102287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.124750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.154955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.167592] LNet: 61053:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.174032] LNet: 61053:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1072.189280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.210338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.235682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.280911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.314331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.352741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.383233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1072.797340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1072.828942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_error -r 1 [ 1072.866262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1072.900340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_error -r 1 [ 1072.931264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1072.964561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_error -r 1 [ 1072.997789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1073.027841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_error -r 1 [ 1073.065291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1073.251135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.274347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.295952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.319557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.344825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.370045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.399530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.428164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.456701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.492580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.523368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.554539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.583188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.620869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.649118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1077.077676] LNet: 61254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.082818] LNetError: 61254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.090731] LNet: Removed LNI 192.168.204.25@tcp [ 1077.101155] LNet: Removed LNI 192.168.204.25@tcp1 [ 1077.365711] Key type .llcrypt unregistered [ 1077.367744] Key type ._llcrypt unregistered [ 1086.980620] Key type ._llcrypt registered [ 1086.983882] Key type .llcrypt registered [ 1087.834458] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:10:47 (1786353047) [ 1088.398459] Key type .llcrypt unregistered [ 1088.400692] Key type ._llcrypt unregistered [ 1091.921460] Key type ._llcrypt registered [ 1091.923745] Key type .llcrypt registered [ 1092.038935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.050144] alg: No test for adler32 (adler32-zlib) [ 1092.911502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1092.935884] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1097.768148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1098.224160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1098.250118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1098.272683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1098.284609] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1098.414595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.433599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.456074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.474771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.495939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.519124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.542595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.569664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.584204] LNet: 62507:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.599384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.621874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.642939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.671950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.696763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.724709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.747761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.073732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1099.099701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1099.130366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1099.148594] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1099.275631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.294477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.312266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.330078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.352372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.383248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.411581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.439441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.458211] LNet: 62616:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.473847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.503417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.522508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.543442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.571875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.598253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.622856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.976884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1100.009652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1100.040842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1100.202131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.228435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.248687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.273843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.298570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.321028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.344930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.373805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.400992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.421253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.446334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.472935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.509906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.533967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.559153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.815186] LNet: 62771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.821484] LNetError: 62771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.836676] LNet: Removed LNI 192.168.204.25@tcp [ 1101.137941] Key type .llcrypt unregistered [ 1101.140560] Key type ._llcrypt unregistered [ 1110.780873] Key type ._llcrypt registered [ 1110.783155] Key type .llcrypt registered [ 1111.693635] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:11:11 (1786353071) [ 1112.207532] Key type .llcrypt unregistered [ 1112.209170] Key type ._llcrypt unregistered [ 1115.732907] Key type ._llcrypt registered [ 1115.734858] Key type .llcrypt registered [ 1115.891807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.905307] alg: No test for adler32 (adler32-zlib) [ 1116.804449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1116.837714] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1122.090494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1128.363117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.422180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1128.448044] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1129.767656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1129.804196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 1 [ 1129.841635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1129.898448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 1 [ 1129.934775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1129.969293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 1 [ 1130.002989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1130.041472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 1 [ 1130.081101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1130.109740] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1130.115924] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1130.332459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.363724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.413287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.460993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.509230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.566621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.611394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.648792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.671287] LNet: 64239:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.701484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.736482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.771211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.804548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.843884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.889882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.928121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.977766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.485347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1131.513189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_error -r 1 [ 1131.539328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1131.578421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_error -r 1 [ 1131.607458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1131.640130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_error -r 1 [ 1131.674954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1131.713948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_error -r 1 [ 1131.756972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1131.783087] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1131.787623] LNet: Skipped 2 previous similar messages [ 1131.998091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.033219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.060250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.087660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.111272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.136580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.167857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.200453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.218035] LNet: 64362:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.234802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.261489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.289692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.314355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.342957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.376469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.400710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.436908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.920475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1132.968819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_timeout -r 1 [ 1133.004326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1133.034967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_timeout -r 1 [ 1133.074263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1133.116064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_timeout -r 1 [ 1133.152639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1133.183784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_timeout -r 1 [ 1133.242251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1133.268237] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1133.273563] LNet: Skipped 1 previous similar message [ 1133.515684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.550174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.576847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.607269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.651524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.700334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.724364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.750386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.769511] LNet: 64485:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.790320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.820392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.852125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.886708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.917746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.945930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.977879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.018755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.965762] LNet: 64567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.971727] LNetError: 64567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.976972] LNet: Removed LNI 192.168.204.25@tcp [ 1137.984742] LNet: Removed LNI 192.168.204.25@tcp1 [ 1138.287590] Key type .llcrypt unregistered [ 1138.290672] Key type ._llcrypt unregistered [ 1149.670235] Key type ._llcrypt registered [ 1149.671828] Key type .llcrypt registered [ 1150.646839] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:11:50 (1786353110) [ 1151.793519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1153.411339] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:11:52 (1786353112) [ 1154.178614] Key type .llcrypt unregistered [ 1154.181660] Key type ._llcrypt unregistered [ 1158.553872] Key type ._llcrypt registered [ 1158.555682] Key type .llcrypt registered [ 1158.711239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.727821] alg: No test for adler32 (adler32-zlib) [ 1159.601347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.628832] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1164.869567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1165.485264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1165.539726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1165.588496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1165.618560] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1165.791646] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1165.831657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.854863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.882258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.915132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.953807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.986552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.015889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.049783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.068563] LNet: 66112:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.088188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.122373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.161939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.195706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.223684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.258327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.579385] LNet: 66155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.585386] LNetError: 66155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.605770] LNet: Removed LNI 192.168.204.25@tcp [ 1166.882874] Key type .llcrypt unregistered [ 1166.884993] Key type ._llcrypt unregistered [ 1179.372979] Key type ._llcrypt registered [ 1179.375609] Key type .llcrypt registered [ 1179.519672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.530433] alg: No test for adler32 (adler32-zlib) [ 1180.427819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1180.451286] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1185.257748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1191.507868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.548447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1191.585308] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1192.790834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1192.843646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e network_timeout -r 1 [ 1192.880615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1192.909741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e network_timeout -r 1 [ 1192.946306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1192.971850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e network_timeout -r 1 [ 1193.004241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1193.040567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e network_timeout -r 1 [ 1193.094402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1193.109860] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1193.267762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.298185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.326710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.351379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.377435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.398563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.417336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.442405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.459805] LNet: 67036:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.473637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.499803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.524638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.554479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.583520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.609480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.846076] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1197.094443] LNet: 67113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.099444] LNetError: 67113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.104854] LNet: Removed LNI 192.168.204.25@tcp [ 1197.116357] LNet: Removed LNI 192.168.204.25@tcp1 [ 1197.372605] Key type .llcrypt unregistered [ 1197.375713] Key type ._llcrypt unregistered [ 1209.234512] Key type ._llcrypt registered [ 1209.236742] Key type .llcrypt registered [ 1210.171357] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:12:49 (1786353169) [ 1210.516233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.530864] alg: No test for adler32 (adler32-zlib) [ 1211.398777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.440506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1211.458342] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1212.040448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.064428] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1212.707434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1212.755554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1212.785775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1212.807648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1212.835618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1212.861675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1212.890717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1212.902442] LNetError: 68212:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1213.471325] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1213.471813] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1215.519750] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1216.228829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1219.615717] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1220.570311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1220.591407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.627255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1220.640334] LNet: 68329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.651498] LNetError: 68329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.658078] LNet: Removed LNI 192.168.204.25@tcp [ 1220.664702] LNet: Removed LNI 192.168.204.25@tcp1 [ 1220.677833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.724791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1220.740734] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1221.232634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.245436] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1221.771443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1221.805613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.833224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1221.860926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1221.879956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1221.908395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1221.934770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1221.968948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1221.988910] LNetError: 68601:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1222.751583] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1225.277334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1228.896084] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1228.900302] LNet: Skipped 1 previous similar message [ 1233.900977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1233.923747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.946696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.310857] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:13:18 (1786353198) [ 1238.402233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.420876] LNet: 69325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.426584] LNetError: 69325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.430922] LNetError: 69325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1238.438495] LNet: Removed LNI 192.168.204.25@tcp [ 1238.447667] LNet: Removed LNI 192.168.204.25@tcp1 [ 1238.466577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.507930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.524184] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1238.970138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.982779] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1239.552473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1239.595626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1239.615994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1239.638223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1239.655879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1239.673820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1239.692641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 0 [ 1239.762165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1240.543243] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [ 1240.544880] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1240.557162] LNet: Skipped 1 previous similar message [ 1255.605582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1255.677149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.697034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.714853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 500 [ 1255.764399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1262.114365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1262.164463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1262.175583] LNet: 69827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.179195] LNetError: 69827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.184145] LNetError: 69827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1262.188287] LNet: Removed LNI 192.168.204.25@tcp [ 1262.201805] LNet: Removed LNI 192.168.204.25@tcp1 [ 1262.213851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.254904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.278801] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1262.768344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1262.783887] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1263.441577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1263.467750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.516613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1263.539604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1263.563164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1263.588567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1263.615589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1263.637463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 0 [ 1264.287938] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1264.292619] LNet: Skipped 3 previous similar messages [ 1264.839138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1275.592470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1275.624660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.663489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.425460] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:14:00 (1786353240) [ 1280.770707] LNet: 70883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.776886] LNetError: 70883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.783370] LNetError: 70883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1280.787399] LNet: Removed LNI 192.168.204.25@tcp [ 1280.794179] LNet: Removed LNI 192.168.204.25@tcp1 [ 1281.285897] Key type .llcrypt unregistered [ 1281.289257] Key type ._llcrypt unregistered [ 1285.508972] Key type ._llcrypt registered [ 1285.512836] Key type .llcrypt registered [ 1285.644159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.652692] alg: No test for adler32 (adler32-zlib) [ 1286.507623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.527469] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1292.721743] Lustre: DEBUG MARKER: Initial discovery [ 1292.762621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1294.324748] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1294.384641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1294.416880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1294.433694] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1294.452493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1294.482624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.125@tcp -r 1 -m GET -l 3 [ 1294.504859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.125@tcp -r 1 -m GET -e local_error [ 1294.529573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.125@tcp -r 1 -m PUT -l 6 [ 1294.751102] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1295.420625] Lustre: DEBUG MARKER: Force oleg425-client.virtnet to discover 192.168.204.125@tcp (in background) [ 1295.485475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1296.353277] Lustre: DEBUG MARKER: Force oleg425-server to discover 192.168.204.25@tcp [ 1297.936790] Lustre: DEBUG MARKER: Wait for 71645 [ 1298.527467] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1299.476375] Lustre: DEBUG MARKER: Finished wait on 71645 [ 1299.554368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1299.588420] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1299.598227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1299.785406] LNet: 72101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.790127] LNetError: 72101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.840622] LNet: Removed LNI 192.168.204.25@tcp [ 1301.090491] Key type .llcrypt unregistered [ 1301.092688] Key type ._llcrypt unregistered [ 1311.202832] Key type ._llcrypt registered [ 1311.204646] Key type .llcrypt registered [ 1312.014763] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:14:31 (1786353271) [ 1312.480703] Key type .llcrypt unregistered [ 1312.482665] Key type ._llcrypt unregistered [ 1316.010625] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1316.116334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1316.119259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1316.122184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1318.469634] Key type ._llcrypt registered [ 1318.472944] Key type .llcrypt registered [ 1318.585805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.597930] alg: No test for adler32 (adler32-zlib) [ 1319.451683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.487455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.505149] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1319.961387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1319.975584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1325.010184] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:14:45 (1786353285) [ 1325.108483] LNet: 74180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.115800] LNetError: 74180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1325.121038] LNet: Removed LNI 10.1.2.3@tcp [ 1325.128387] LNet: Removed LNI 192.168.204.25@tcp [ 1325.552841] Key type .llcrypt unregistered [ 1325.554583] Key type ._llcrypt unregistered [ 1329.216263] Key type ._llcrypt registered [ 1329.218366] Key type .llcrypt registered [ 1329.335650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1329.360459] alg: No test for adler32 (adler32-zlib) [ 1330.218392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.262194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1330.276164] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1330.712796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1330.725845] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1331.263808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.25@tcp [ 1332.320134] LNetError: Unexpected error -22 connecting to 192.168.204.25@tcp at host 192.168.204.25:988 [ 1334.989292] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:14:55 (1786353295) [ 1335.096760] LNet: 75313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.110942] LNetError: 75313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1335.117278] LNet: Removed LNI 10.1.2.3@tcp [ 1335.131306] LNet: Removed LNI 192.168.204.25@tcp [ 1335.606462] Key type .llcrypt unregistered [ 1335.609914] Key type ._llcrypt unregistered [ 1339.005747] Key type ._llcrypt registered [ 1339.008413] Key type .llcrypt registered [ 1339.166195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.182423] alg: No test for adler32 (adler32-zlib) [ 1340.037514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.084869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.104286] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1340.590180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1340.604925] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1341.133219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.25@tcp1 --nid 192.168.204.25@tcp2 [ 1343.425615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp1 192.168.204.25@tcp2 [ 1343.437258] LNetError: 76159:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.25@tcp1 to 192.168.204.25@tcp2 - Route table may be misconfigured [ 1343.453994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp2 192.168.204.25@tcp1 [ 1346.919916] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:15:06 (1786353306) [ 1346.963510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.979666] LNet: 76724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.985753] LNetError: 76724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.991183] LNet: Removed LNI 192.168.204.25@tcp1 [ 1348.005806] LNet: Removed LNI 192.168.204.25@tcp2 [ 1348.023263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.060853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.078839] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1348.553317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.566706] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1349.102587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1349.127806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.25@tcp [ 1349.153651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp1 192.168.204.25@tcp1 [ 1349.182593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.25@tcp -d 192.168.204.25@tcp -e network_timeout [ 1349.203808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -e network_timeout [ 1349.225806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -e network_timeout [ 1349.260326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -e network_timeout [ 1349.282772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1349.297957] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1349.313610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1349.380099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1349.443085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1350.111319] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1353.330945] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:15:13 (1786353313) [ 1353.433971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.454516] LNet: 77593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.460592] LNetError: 77593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.464864] LNetError: 77593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1353.469497] LNet: Removed LNI 192.168.204.25@tcp [ 1354.533904] LNet: Removed LNI 192.168.204.25@tcp1 [ 1354.547429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.628484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1354.813385] LNet: 77635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.012597] Key type .llcrypt unregistered [ 1355.014341] Key type ._llcrypt unregistered [ 1360.800479] Key type ._llcrypt registered [ 1360.801872] Key type .llcrypt registered [ 1361.519767] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:15:21 (1786353321) [ 1361.726517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.736985] alg: No test for adler32 (adler32-zlib) [ 1362.589985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.475719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1364.505669] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:15:24 (1786353324) [ 1364.553223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.564820] LNet: 78724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.587514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.634374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.650488] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1365.063386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1365.078724] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1365.662762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1365.685629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp1 [ 1365.711758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp1 [ 1369.559326] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:15:29 (1786353329) [ 1370.639899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg425-server [ 1371.592967] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:15:31 (1786353331) [ 1372.611268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg425-server [ 1373.572104] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:15:33 (1786353333) [ 1374.560139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg425-server [ 1375.492868] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:15:35 (1786353335) [ 1376.647103] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg425-server [ 1377.575705] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:15:37 (1786353337) [ 1378.841227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg425-server [ 1379.889131] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:15:39 (1786353339) [ 1381.008668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg425-server [ 1382.034418] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:15:42 (1786353342) [ 1383.109234] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg425-server [ 1384.132235] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:15:44 (1786353344) [ 1385.215263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg425-server [ 1386.278031] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:15:46 (1786353346) [ 1387.319164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg425-server [ 1388.278466] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:15:48 (1786353348) [ 1388.352477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.367326] LNet: 82604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.378285] LNetError: 82604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.382638] LNet: Removed LNI 192.168.204.25@tcp [ 1389.414204] LNet: Removed LNI 192.168.204.25@tcp1 [ 1389.425047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.476967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.492477] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1389.921485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1389.961387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1391.137267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.151048] LNet: 82763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.155428] LNetError: 82763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.161202] LNetError: 82763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1391.171715] LNet: Removed LNI 192.168.204.25@tcp [ 1391.183705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.217876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1391.233908] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1391.736771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1391.859248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.933743] LNet: Removed LNI 192.168.204.25@tcp [ 1392.951443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.000730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.016961] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1393.475416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1393.535377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1393.679791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.692513] LNet: 83056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.696865] LNet: 83056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1393.700963] LNetError: 83056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.704833] LNetError: 83056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1393.719560] LNet: Removed LNI 192.168.204.25@tcp [ 1393.731894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.766942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1393.783201] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1394.326515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1394.461927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.494160] LNet: Removed LNI 192.168.204.25@tcp [ 1395.511829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.566861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1395.585479] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1396.064670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1396.120720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1396.242281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.260827] LNetError: 83348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.266089] LNetError: 83348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1396.279350] LNet: Removed LNI 192.168.204.25@tcp [ 1396.290429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.319763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1396.336154] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1396.784185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1397.969719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.986627] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.991674] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1399.013573] LNet: Removed LNI 192.168.204.25@tcp [ 1399.033903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.074447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.094053] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1399.571121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1399.624892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1399.805521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.869773] LNet: Removed LNI 192.168.204.25@tcp [ 1400.896360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.947458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1400.982331] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1401.544127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1401.719969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.736517] LNetError: 83795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.740356] LNetError: 83795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1401.756118] LNet: Removed LNI 192.168.204.25@tcp [ 1401.774912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.818396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.838620] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1402.331538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1402.389498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1402.550226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.622270] LNet: Removed LNI 192.168.204.25@tcp [ 1403.641411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.684716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1403.710487] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1404.202100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1404.375530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1405.414723] LNet: Removed LNI 192.168.204.25@tcp [ 1405.448291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.493950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1405.512747] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1405.958579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1406.006781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1407.239397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.253315] LNet: 84244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.257309] LNet: 84244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1408.293398] LNet: Removed LNI 192.168.204.25@tcp [ 1408.307460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.345393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1408.363850] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1408.834679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1410.100382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.118673] LNetError: 84398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.129910] LNetError: 84398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1410.140460] LNet: Removed LNI 192.168.204.25@tcp [ 1410.163900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.215172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.238853] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1410.714948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1410.752193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.774522] LNet: Removed LNI 192.168.204.25@tcp [ 1410.787597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.828837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.844543] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1411.373221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1411.426771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.447747] LNet: Removed LNI 192.168.204.25@tcp [ 1411.460142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.491390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1411.530983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.550871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.581829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1411.641537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.665561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.700880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1411.758478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.783896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.825976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1411.857107] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1412.419634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.440787] LNet: Removed LNI 192.168.204.25@tcp [ 1412.453934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.500639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1416.966206] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:16:16 (1786353376) [ 1417.140827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.165752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.218882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1417.237939] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1417.989455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.020295] LNet: Removed LNI 192.168.204.25@tcp231 [ 1418.037383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.096561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1418.115744] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1418.120347] LNet: 85564:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1418.400634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1418.436400] LNet: Removed LNI 192.168.204.25@tcp231 [ 1418.451654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1418.471929] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1419.230241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.255273] LNet: Removed LNI 192.168.204.25@tcp231 [ 1419.267659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.317359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1419.338282] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/0] [ 1419.344344] LNet: 85716:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1419.664414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1419.691732] LNet: Removed LNI 192.168.204.25@tcp231 [ 1419.711093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1419.732323] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/0] [ 1420.507313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.531383] LNet: Removed LNI 192.168.204.25@tcp231 [ 1420.548611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.605378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1420.628533] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/60] [ 1420.634810] LNet: 85867:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1420.861216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1420.881650] LNet: Removed LNI 192.168.204.25@tcp231 [ 1420.891801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1420.911807] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/60] [ 1421.714766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1421.736741] LNet: Removed LNI 192.168.204.25@tcp231 [ 1421.749849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.792654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1421.815620] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1422.090737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1422.116396] LNet: Removed LNI 192.168.204.25@tcp231 [ 1422.136744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1422.162939] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1426.781830] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:16:26 (1786353386) [ 1426.846443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.874604] LNet: 86723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.879908] LNet: 86723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 1426.886029] LNetError: 86723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.891155] LNetError: 86723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1426.903843] LNet: Removed LNI 192.168.204.25@tcp231 [ 1426.922274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.992460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1427.014221] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1427.508823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1427.535577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1427.562159] LNet: Removed LNI 192.168.204.25@tcp232 [ 1427.576502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1427.598915] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1428.114618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1428.146615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1428.173059] LNet: Removed LNI 192.168.204.25@tcp232 [ 1428.188888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1428.213990] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1428.692648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1428.720303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1428.743201] LNet: Removed LNI 192.168.204.25@tcp232 [ 1428.757269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1428.781094] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1429.314848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1429.351165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1429.377785] LNet: Removed LNI 192.168.204.25@tcp232 [ 1429.390320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1433.557600] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:16:33 (1786353393) [ 1434.346692] Key type .llcrypt unregistered [ 1434.348610] Key type ._llcrypt unregistered [ 1438.187467] Key type ._llcrypt registered [ 1438.190671] Key type .llcrypt registered [ 1438.333264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1438.347812] alg: No test for adler32 (adler32-zlib) [ 1439.207868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1439.226221] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1443.995564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1449.036482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.067759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.085273] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1450.106984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 2 [ 1450.135727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 2 [ 1450.159792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 2 [ 1450.183721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 2 [ 1450.214264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_interrupt -r 2 [ 1450.243814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 2 [ 1450.271243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_interrupt -r 2 [ 1450.300700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 2 [ 1450.331927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1450.346615] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1450.350788] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1450.516703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1450.537351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1450.559236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1450.580126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1450.605938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1450.634314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1450.660769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1450.699327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1450.715461] LNet: 88521:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1450.728823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1450.754203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1450.777254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1450.800380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1450.839097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1450.860325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1451.209257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 2 [ 1451.243133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 2 [ 1451.265545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 2 [ 1451.298925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 2 [ 1451.329673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_dropped -r 2 [ 1451.355353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 2 [ 1451.380418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_dropped -r 2 [ 1451.404412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 2 [ 1451.429810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1451.449639] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1451.595394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.617406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1451.640721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.661455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.683922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.718649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.748176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.775512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.790275] LNet: 88641:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.807247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1451.826893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1451.848813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.877149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.902258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1451.937236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1452.354954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 2 [ 1452.389190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 2 [ 1452.424714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 2 [ 1452.454623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 2 [ 1452.481800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_aborted -r 2 [ 1452.508329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 2 [ 1452.536047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_aborted -r 2 [ 1452.563231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 2 [ 1452.587180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1452.599521] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1452.732229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1452.751341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1452.775402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1452.797270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1452.818125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1452.836870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1452.858661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1452.881893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1452.897379] LNet: 88762:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1452.909917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1452.935645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1452.960365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.978775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.997721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.021170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.335921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 2 [ 1453.362920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 2 [ 1453.394775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 2 [ 1453.424227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 2 [ 1453.457840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_no_route -r 2 [ 1453.484393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 2 [ 1453.508792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_no_route -r 2 [ 1453.537545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 2 [ 1453.564412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1453.739129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1453.765327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1453.789566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1453.810392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1453.841709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1453.866756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1453.887216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1453.904278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1453.926678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1453.951140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.972747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.997786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1454.018592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1454.042733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1454.342367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 2 [ 1454.363489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 2 [ 1454.388648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 2 [ 1454.415557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 2 [ 1454.437166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_timeout -r 2 [ 1454.464256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 2 [ 1454.485630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_timeout -r 2 [ 1454.508854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 2 [ 1454.534641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1454.665315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1454.689152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1454.707681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1454.727847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1454.747984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1454.770592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1454.790697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1454.819486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1454.842216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1454.865422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1454.885486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1454.908327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1454.933639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1454.959448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1455.267710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 2 [ 1455.290269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 2 [ 1455.315856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 2 [ 1455.345749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 2 [ 1455.367533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_dropped -r 2 [ 1455.396499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 2 [ 1455.416715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_dropped -r 2 [ 1455.450043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 2 [ 1455.480567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1455.502769] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1455.506599] LNet: Skipped 3 previous similar messages [ 1455.509497] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1455.702919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1455.730726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1455.756812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.778710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.800287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.824572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.848814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.874644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.887612] LNet: 89121:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.895263] LNet: 89121:0:(api-ni.c:295:recovery_interval_set()) Skipped 2 previous similar messages [ 1455.909227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1455.926946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1455.948544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.974268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1455.998566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1456.019369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1459.123462] LNet: 89200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.127356] LNetError: 89200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.133543] LNet: Removed LNI 192.168.204.25@tcp [ 1459.146550] LNet: Removed LNI 192.168.204.25@tcp1 [ 1459.362532] Key type .llcrypt unregistered [ 1459.364816] Key type ._llcrypt unregistered [ 1467.639140] Key type ._llcrypt registered [ 1467.640861] Key type .llcrypt registered [ 1468.530380] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:17:08 (1786353428) [ 1468.778988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1468.787775] alg: No test for adler32 (adler32-zlib) [ 1469.642390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.693862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.724143] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1470.351534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1470.377754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1470.392652] LNetError: 90177:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1470.469937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.25@tcp [ 1470.570397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1475.167902] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:17:15 (1786353435) [ 1475.267985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.286724] LNet: 90766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.299931] LNetError: 90766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.319534] LNet: Removed LNI 192.168.204.25@tcp [ 1475.336327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.385452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1475.404613] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1475.982200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1476.007149] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1476.838861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.25@tcp [ 1476.858129] LNetError: 91033:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1477.407384] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1477.408819] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1477.999211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.25@tcp [ 1484.528463] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:17:24 (1786353444) [ 1485.948752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg425-server [ 1487.173980] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:17:26 (1786353446) [ 1488.511055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg425-server [ 1489.654082] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:17:29 (1786353449) [ 1490.958263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg425-server [ 1492.268530] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:17:31 (1786353451) [ 1492.333365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.351550] LNet: 92657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.357954] LNetError: 92657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.362170] LNet: Removed LNI 192.168.204.25@tcp [ 1492.370602] LNet: Removed LNI 192.168.204.25@tcp1 [ 1492.388062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.438815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.461586] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1493.001404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1493.046133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1493.078631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1493.131295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1493.170766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1493.202611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1497.255666] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:17:37 (1786353457) [ 1497.356887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.374591] LNet: 93385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.380995] LNetError: 93385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.389955] LNetError: 93385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1497.414106] LNet: Removed LNI 192.168.204.25@tcp [ 1497.432882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.473628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.490341] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1501.828940] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:17:41 (1786353461) [ 1501.894954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.912753] LNet: 94084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.919819] LNetError: 94084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.934298] LNet: Removed LNI 192.168.204.25@tcp [ 1501.952235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.997383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.016187] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1506.832264] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:17:46 (1786353466) [ 1507.882456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1509.153081] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:17:48 (1786353468) [ 1509.445524] LNet: 95126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.449383] LNetError: 95126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.467056] LNet: Removed LNI 192.168.204.25@tcp [ 1509.910752] Key type .llcrypt unregistered [ 1509.913718] Key type ._llcrypt unregistered [ 1513.401893] Key type ._llcrypt registered [ 1513.406459] Key type .llcrypt registered [ 1513.528906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1513.541879] alg: No test for adler32 (adler32-zlib) [ 1514.405927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1514.433552] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1519.282678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1521.948689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.125@tcp [ 1522.156590] LNet: 95558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.160690] LNetError: 95558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.170063] LNet: Removed LNI 192.168.204.25@tcp [ 1522.396420] Key type .llcrypt unregistered [ 1522.398385] Key type ._llcrypt unregistered [ 1527.930668] Key type ._llcrypt registered [ 1527.932195] Key type .llcrypt registered [ 1528.581560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1529.405562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1530.254048] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:18:10 (1786353490) [ 1531.165481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1532.104442] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:18:12 (1786353492) [ 1533.103955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg425-server [ 1533.952404] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:18:14 (1786353494) [ 1534.809150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg425-server [ 1535.628053] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:18:15 (1786353495) [ 1536.219307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1536.254303] alg: No test for adler32 (adler32-zlib) [ 1537.170199] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1537.302486] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1538.919279] Key type lgssc registered [ 1539.502946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1550.708944] Key type lgssc unregistered [ 1550.881861] LNet: 98732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.887295] LNetError: 98732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.895114] LNet: Removed LNI 192.168.204.25@tcp [ 1551.261382] Key type .llcrypt unregistered [ 1551.263223] Key type ._llcrypt unregistered [ 1558.752805] Key type ._llcrypt registered [ 1558.754970] Key type .llcrypt registered [ 1559.579574] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:18:39 (1786353519) [ 1559.835908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.845786] alg: No test for adler32 (adler32-zlib) [ 1560.715485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.762409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.789471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.824430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.866954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1564.817470] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:18:44 (1786353524) [ 1564.871064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.881721] LNet: 100140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.900862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.950490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.975708] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1565.536833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.550766] LNet: 100289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.555429] LNetError: 100289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.564548] LNet: Removed LNI 192.168.204.25@tcp [ 1565.581501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.613420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1565.626900] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1565.919479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.947389] LNet: Removed LNI 192.168.204.25@tcp [ 1565.959841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.990885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1566.007363] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1569.492331] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:18:49 (1786353529) [ 1569.545793] LNet: 100899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.549285] LNet: 100899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1569.555206] LNetError: 100899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.559295] LNetError: 100899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.570654] LNet: Removed LNI 192.168.204.25@tcp [ 1569.933480] Key type .llcrypt unregistered [ 1569.934855] Key type ._llcrypt unregistered [ 1572.859298] Key type ._llcrypt registered [ 1572.860644] Key type .llcrypt registered [ 1572.960967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1572.969956] alg: No test for adler32 (adler32-zlib) [ 1573.842599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.895185] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1573.897438] LNet: *** cfs_fail_loc=e003, val=100*** [ 1574.318903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1574.338655] Ping reply received with mapping: [ 1574.340897] found metadata reply NI[192.168.204.25@tcp] [ 1574.343285] 19 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.345686] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.348466] 5a 5a 5a 5a 5a 5a 5a 5a [ 1574.442742] LNet: 101307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.449193] LNetError: 101307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.463556] LNet: Removed LNI 192.168.204.25@tcp [ 1574.816491] Key type .llcrypt unregistered [ 1574.818478] Key type ._llcrypt unregistered [ 1579.934920] Key type ._llcrypt registered [ 1579.936787] Key type .llcrypt registered [ 1580.545792] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:19:00 (1786353540) [ 1580.798289] Key type .llcrypt unregistered [ 1580.800082] Key type ._llcrypt unregistered [ 1583.641700] Key type ._llcrypt registered [ 1583.643132] Key type .llcrypt registered [ 1583.748506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1583.757382] alg: No test for adler32 (adler32-zlib) [ 1584.720542] LNetError: 102363:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1584.959415] Key type .llcrypt unregistered [ 1584.964200] Key type ._llcrypt unregistered [ 1587.691290] Key type ._llcrypt registered [ 1587.693148] Key type .llcrypt registered [ 1588.481509] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:19:08 (1786353548) [ 1588.802621] Key type .llcrypt unregistered [ 1588.804349] Key type ._llcrypt unregistered [ 1591.895566] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1591.897321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1594.234295] Key type ._llcrypt registered [ 1594.236494] Key type .llcrypt registered [ 1594.327541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.337372] alg: No test for adler32 (adler32-zlib) [ 1595.196813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.233289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1600.955594] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:19:20 (1786353560) [ 1601.018358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.029757] LNet: 104148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.033838] LNetError: 104148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1601.045197] LNet: Removed LNI 10.1.2.3@tcp [ 1601.061716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.444730] Key type .llcrypt unregistered [ 1601.446794] Key type ._llcrypt unregistered [ 1604.820497] Key type ._llcrypt registered [ 1604.823911] Key type .llcrypt registered [ 1604.994840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1605.005836] alg: No test for adler32 (adler32-zlib) [ 1605.882893] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1605.935723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1606.058403] LNet: 104441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.063647] LNetError: 104441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.076305] LNet: Removed LNI 192.168.204.25@tcp [ 1606.636729] Key type .llcrypt unregistered [ 1606.638939] Key type ._llcrypt unregistered [ 1613.765774] Key type ._llcrypt registered [ 1613.767510] Key type .llcrypt registered [ 1614.629857] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:19:34 (1786353574) [ 1615.057477] Key type .llcrypt unregistered [ 1615.060136] Key type ._llcrypt unregistered [ 1619.154266] Key type ._llcrypt registered [ 1619.157843] Key type .llcrypt registered [ 1619.317767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.329883] alg: No test for adler32 (adler32-zlib) [ 1623.978000] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:19:43 (1786353583) [ 1624.123380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.144835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.197619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.215444] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1624.750773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.778257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1624.802992] LNet: Removed LNI 192.168.204.25@tcp [ 1625.001682] LNet: 106195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.292443] Key type .llcrypt unregistered [ 1625.295322] Key type ._llcrypt unregistered [ 1632.240069] Key type ._llcrypt registered [ 1632.241421] Key type .llcrypt registered [ 1633.113238] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:19:52 (1786353592) [ 1633.364419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.372712] alg: No test for adler32 (adler32-zlib) [ 1634.218949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.263542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.286122] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1634.750260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1634.775163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.25@tcp [ 1634.790728] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1634.793836] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1634.799355] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1634, 0, 0 [ 1634.803714] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1634.806122] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1634.808807] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1634.811867] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1634, 0, 0 [ 1634.816216] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1634.818294] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1634.820596] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1634.823559] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1634, 0, 1684 [ 1634.827496] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1634.829709] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1634.832648] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1634.835168] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1634, 0, 1684 [ 1634.839293] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1638.556181] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:19:58 (1786353598) [ 1638.965956] LNet: 107734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.972790] LNetError: 107734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.987653] LNet: Removed LNI 192.168.204.25@tcp [ 1639.392805] Key type .llcrypt unregistered [ 1639.396747] Key type ._llcrypt unregistered [ 1642.707073] Key type ._llcrypt registered [ 1642.708917] Key type .llcrypt registered [ 1642.830752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1642.839763] alg: No test for adler32 (adler32-zlib) [ 1643.683857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1643.706891] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1648.584178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1653.155768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.185127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1653.208376] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1653.803629] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1653.863144] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1653.869356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1653.871956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1656.085962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1656.105405] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1661.894342] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:20:22 (1786353622) [ 1661.975181] LNet: 109234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.978975] LNetError: 109234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.984241] LNet: Removed LNI 192.168.204.25@tcp [ 1663.007146] LNet: Removed LNI 192.168.204.25@tcp1 [ 1663.008914] LNetError: 109234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1663.012095] LNetError: 109234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1663.023566] LNet: Removed LNI 10.1.2.3@tcp99 [ 1663.373818] Key type .llcrypt unregistered [ 1663.378442] Key type ._llcrypt unregistered [ 1666.351696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1666.355264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1668.637641] Key type ._llcrypt registered [ 1668.639405] Key type .llcrypt registered [ 1668.747748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1668.755923] alg: No test for adler32 (adler32-zlib) [ 1669.582678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.610268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.622909] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1670.001703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1670.016236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1671.128916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1671.150988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.176191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1671.191675] LNetError: 109986:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1671.215642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.244923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1671.465883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1671.500598] LNet: 110006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.505262] LNetError: 110006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1671.509843] LNet: Removed LNI 10.1.2.3@tcp [ 1672.549517] LNet: Removed LNI 192.168.204.25@tcp [ 1672.906585] Key type .llcrypt unregistered [ 1672.912208] Key type ._llcrypt unregistered [ 1679.035412] Key type ._llcrypt registered [ 1679.038441] Key type .llcrypt registered [ 1679.831216] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:20:39 (1786353639) [ 1680.158455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.168736] alg: No test for adler32 (adler32-zlib) [ 1681.016475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.055429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.075624] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1681.556135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1681.606731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg425-client@tcp [ 1685.384296] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:20:45 (1786353645) [ 1685.456789] LNet: 111539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.462721] LNetError: 111539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.476738] LNet: Removed LNI 192.168.204.25@tcp [ 1685.862537] Key type .llcrypt unregistered [ 1685.863928] Key type ._llcrypt unregistered [ 1689.338928] Key type ._llcrypt registered [ 1689.340984] Key type .llcrypt registered [ 1689.458753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.471282] alg: No test for adler32 (adler32-zlib) [ 1690.301828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1690.318507] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1690.338984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1693.725433] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:20:53 (1786353653) [ 1694.509873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1695.497258] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:20:55 (1786353655) [ 1696.320703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1697.431887] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:20:57 (1786353657) [ 1697.574575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.587705] LNet: 112963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.591984] LNetError: 112963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.608924] LNet: Removed LNI 192.168.204.25@tcp [ 1697.624184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.652528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.668965] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1698.150207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1698.178923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1702.405731] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:21:02 (1786353662) [ 1702.540865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.553765] LNet: 113672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.559905] LNetError: 113672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.569122] LNet: Removed LNI 192.168.204.25@tcp [ 1702.583764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.621482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1702.644370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1703.122307] Key type .llcrypt unregistered [ 1703.126764] Key type ._llcrypt unregistered [ 1710.273340] Key type ._llcrypt registered [ 1710.274585] Key type .llcrypt registered [ 1711.051623] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:21:11 (1786353671) [ 1711.361295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.369835] alg: No test for adler32 (adler32-zlib) [ 1712.207504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.240961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.258966] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1712.711194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1712.751413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1717.056852] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:21:17 (1786353677) [ 1717.168931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.185500] LNet: 115231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.193159] LNetError: 115231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.203614] LNet: Removed LNI 192.168.204.25@tcp [ 1717.220059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.259741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1717.284769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1720.845440] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:21:20 (1786353680) [ 1720.972755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.987709] LNet: 115811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.003918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.041184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1721.247547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.272398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1721.305938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1721.518317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.548220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1721.595739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.618478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.663675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1721.855373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.881464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1721.920263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1722.109930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1722.135557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1725.709795] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:21:25 (1786353685) [ 1725.773321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.794503] LNet: 116438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.798906] LNet: 116438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1725.814528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.849955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.873884] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1726.349761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1726.365290] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1726.992162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1727.035857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1727.181433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.198864] LNetError: 116724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.202846] LNet: Removed LNI 192.168.204.25@tcp [ 1727.210860] LNet: Removed LNI 192.168.204.25@tcp2 [ 1727.225397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.261306] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1727.265158] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1727.272595] LNet: 116730:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.542803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1727.670912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.735792] LNet: Removed LNI 192.168.204.25@tcp [ 1728.737362] LNetError: 116767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.740930] LNetError: 116767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1728.752859] LNet: Removed LNI 192.168.204.25@tcp2 [ 1728.767614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.811889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.828745] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1729.257506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1729.269261] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1729.769990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1729.790819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1729.824116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1729.949616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.962182] LNet: 117052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.967166] LNet: 117052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1730.975239] LNet: Removed LNI 192.168.204.25@tcp [ 1730.978843] LNetError: 117052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.987040] LNetError: 117052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1730.999208] LNet: Removed LNI 192.168.204.25@tcp2 [ 1731.020807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.070403] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1731.073838] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1731.077582] LNet: 117058:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.369477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1731.511992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.523315] LNet: Removed LNI 192.168.204.25@tcp [ 1732.581462] LNet: Removed LNI 192.168.204.25@tcp2 [ 1732.596622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.635844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.650708] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1733.130850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1733.144268] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1733.680959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1733.724737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1733.877079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.943161] LNet: Removed LNI 192.168.204.25@tcp [ 1734.953417] LNet: Removed LNI 192.168.204.25@tcp2 [ 1734.970775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.020753] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1735.026485] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1735.031419] LNet: 117381:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1735.315895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1735.421048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.433796] LNet: 117418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.436978] LNet: 117418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1735.441379] LNetError: 117418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.444994] LNetError: 117418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1735.448890] LNet: Removed LNI 192.168.204.25@tcp [ 1735.456891] LNet: Removed LNI 192.168.204.25@tcp2 [ 1735.472662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.511371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.527660] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1735.936895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1735.947657] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1736.478166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1736.499371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1736.540194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1736.683905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.759144] LNet: Removed LNI 192.168.204.25@tcp [ 1738.788682] LNet: Removed LNI 192.168.204.25@tcp2 [ 1738.803485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.837194] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1738.841413] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1738.845232] LNet: 117709:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1739.085377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1739.205716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.220343] LNet: Removed LNI 192.168.204.25@tcp [ 1740.261210] LNet: Removed LNI 192.168.204.25@tcp2 [ 1740.281816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.327701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.342417] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1740.780232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1740.793277] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1741.316215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1741.334864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1741.371287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1741.540516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.559442] LNet: Removed LNI 192.168.204.25@tcp [ 1742.567652] LNet: Removed LNI 192.168.204.25@tcp2 [ 1742.581321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.617099] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1742.620561] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1742.849403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1743.041315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.095149] LNet: Removed LNI 192.168.204.25@tcp [ 1744.098157] LNetError: 118100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.103390] LNetError: 118100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1744.114744] LNet: Removed LNI 192.168.204.25@tcp2 [ 1744.130578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.167710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.185900] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1744.569503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1744.581025] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1745.069620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1745.094560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1745.110539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1745.141071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1745.326605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.340240] LNet: 118402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.343988] LNet: 118402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1746.399160] LNet: Removed LNI 192.168.204.25@tcp [ 1747.429768] LNet: Removed LNI 192.168.204.25@tcp2 [ 1747.448954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.480250] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1747.484547] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1747.489021] LNet: 118408:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1747.494697] LNet: 118408:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1747.719379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1751.364946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1752.258895] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:21:52 (1786353712) [ 1752.330899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.349707] LNet: Removed LNI 192.168.204.25@tcp [ 1753.383089] LNet: Removed LNI 192.168.204.25@tcp2 [ 1753.403959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.579100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.593828] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1753.998064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1754.016655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1754.552988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.565663] LNet: Removed LNI 10.1.2.3@tcp [ 1754.579436] LNet: Removed LNI 192.168.204.25@tcp [ 1754.595441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.641989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1754.664599] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1754.667561] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1754.883774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1754.896885] LNet: Removed LNI 192.168.204.25@tcp [ 1754.903303] LNet: Removed LNI 10.1.2.3@tcp [ 1758.322084] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 05:21:58 (1786353718) [ 1758.376244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.416028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.451950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.465874] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1758.947889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1758.978891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.001706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.026602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.048809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.067508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.090373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.108513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.132720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.159183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.184194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1759.269862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1762.989278] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:22:02 (1786353722) [ 1763.058549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.071822] LNet: 120774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.079186] LNet: 120774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1763.084109] LNetError: 120774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.088345] LNetError: 120774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1763.103614] LNet: Removed LNI 192.168.204.25@tcp [ 1763.126325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.202801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1763.309467] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1763.314680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1763.319469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1765.518944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.536445] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1765.956429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1765.969985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1767.324533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1767.356290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1767.399264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.25@tcp -m PUT -e local_timeout -r 1 [ 1767.446168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1767.489605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1767.491595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1767.649533] LNetError: 120847:0:(lib-move.c:4801:lnet_parse()) 192.168.204.25@tcp, src 192.168.204.25@tcp: Dropping PUT (error -108 looking up sender) [ 1769.695258] LNet: Removed LNI 10.1.2.3@tcp [ 1770.725337] LNet: Removed LNI 192.168.204.25@tcp [ 1770.998407] Key type .llcrypt unregistered [ 1771.000080] Key type ._llcrypt unregistered [ 1777.210218] Key type ._llcrypt registered [ 1777.211217] Key type .llcrypt registered [ 1777.973830] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:22:17 (1786353737) [ 1778.197553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.210735] alg: No test for adler32 (adler32-zlib) [ 1779.059518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.273953] LNet: 122160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.627683] Key type .llcrypt unregistered [ 1779.629704] Key type ._llcrypt unregistered [ 1782.916179] Key type ._llcrypt registered [ 1782.918365] Key type .llcrypt registered [ 1783.035257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.044370] alg: No test for adler32 (adler32-zlib) [ 1783.929376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1783.958382] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1788.954066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1789.065353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1789.404066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.125@tcp [ 1789.573678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.605606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.625560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.646386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.671400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.687917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.710910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.732320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1789.749420] LNet: 122613:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.770436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1789.793089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1789.814714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1789.835440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1789.861380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1789.886785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.278141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1790.393411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.417384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.438881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.464394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.491252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.519996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.553941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.578248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.591306] LNet: 122731:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.605232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.629159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.654640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.679563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.700356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.728385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.127545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.125@tcp [ 1791.256892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.282903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1791.311432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.340704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.376112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.403514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.428719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.453655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.478579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.501468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.519454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.542540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.565534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.589719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.996357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1792.155694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.160322] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1792.177030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1792.201906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.226208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.248547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.271117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.296958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.327666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.342611] LNet: 122965:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.346839] LNet: 122965:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1792.359197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.382983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.409597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.433326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.467333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.492988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.730532] LNet: 123015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.735281] LNetError: 123015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.747074] LNet: Removed LNI 192.168.204.25@tcp [ 1792.996835] Key type .llcrypt unregistered [ 1792.998590] Key type ._llcrypt unregistered [ 1802.538797] Key type ._llcrypt registered [ 1802.540298] Key type .llcrypt registered [ 1803.448888] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:22:43 (1786353763) [ 1803.671156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.681353] alg: No test for adler32 (adler32-zlib) [ 1804.542956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.781806] LNet: 123902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.140577] Key type .llcrypt unregistered [ 1805.142852] Key type ._llcrypt unregistered [ 1808.324645] Key type ._llcrypt registered [ 1808.326498] Key type .llcrypt registered [ 1808.425708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.433730] alg: No test for adler32 (adler32-zlib) [ 1809.269038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1809.285814] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1813.361388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1817.319531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.337977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1817.350077] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1817.879894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1818.214091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.125@tcp [ 1818.355185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1818.373480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1818.398955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1818.421232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1818.447385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1818.468733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1818.489936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1818.505640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1818.516448] LNet: 124557:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1818.527092] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1818.528892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1818.555779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1818.578363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1818.604476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1818.626888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1818.644441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.965611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1819.089164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.105142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.122732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.139762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.155964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.172734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.190552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.207932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.221163] LNet: 124674:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.233642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.253472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.273636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.291209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.310395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.336838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1819.664911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.125@tcp [ 1819.793907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.812844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.831598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.857127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.875715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.892667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.906867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.923660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.941637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.963838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.982059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1820.001602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1820.019648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1820.041672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.352252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1820.476771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1820.494209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1820.513068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1820.531869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1820.552413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1820.571905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1820.593860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1820.612185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1820.622848] LNet: 124908:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1820.626445] LNet: 124908:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1820.636961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1820.653086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1820.670611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1820.690152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1820.708912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1820.731937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1823.715356] LNet: 124993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.719581] LNetError: 124993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.723270] LNet: Removed LNI 192.168.204.25@tcp [ 1823.729828] LNet: Removed LNI 192.168.204.25@tcp1 [ 1823.978176] Key type .llcrypt unregistered [ 1823.979848] Key type ._llcrypt unregistered [ 1832.639275] Key type ._llcrypt registered [ 1832.640943] Key type .llcrypt registered [ 1833.349502] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 05:23:13 (1786353793) [ 1833.564746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.574733] alg: No test for adler32 (adler32-zlib) [ 1834.392146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.435181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.451187] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1834.821330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1834.835149] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1835.313205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1835.349223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1836.383158] LNet: Removed LNI 192.168.204.25@tcp [ 1839.409439] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 05:23:19 (1786353799) [ 1840.214272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1841.167387] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:23:21 (1786353801) [ 1845.080985] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:23:25 (1786353805) [ 1849.190552] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:23:29 (1786353809) [ 1852.833803] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:23:32 (1786353812) [ 1856.626392] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:23:36 (1786353816) [ 1860.177705] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:23:40 (1786353820) [ 1863.670637] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:23:43 (1786353823) [ 1867.556431] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:23:47 (1786353827) [ 1871.258949] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:23:51 (1786353831) [ 1874.416176] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:23:54 (1786353834) [ 1877.606564] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:23:57 (1786353837) [ 1880.840447] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:24:01 (1786353841) [ 1884.131793] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:24:04 (1786353844) [ 1887.325234] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:24:07 (1786353847) [ 1890.661205] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:24:10 (1786353850) [ 1893.915553] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:24:14 (1786353854) [ 1897.243836] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:24:17 (1786353857) [ 1900.591985] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:24:20 (1786353860) [ 1900.660376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.671971] LNet: 136688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.677145] LNetError: 136688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.687225] LNet: Removed LNI 192.168.204.25@tcp2 [ 1900.700270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.761776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.773491] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1901.302333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.313131] LNet: 136831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.317334] LNetError: 136831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.328542] LNet: Removed LNI 192.168.204.25@tcp [ 1901.338229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.363160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1901.373621] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1904.553648] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:24:24 (1786353864) [ 1907.973641] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:24:28 (1786353868) [ 1911.594873] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:24:31 (1786353871) [ 1914.954192] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:24:35 (1786353875) [ 1918.050357] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:24:38 (1786353878) [ 1921.310961] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:24:41 (1786353881) [ 1924.746547] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:24:44 (1786353884) [ 1928.078282] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:24:48 (1786353888) [ 1931.111447] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:24:51 (1786353891) [ 1934.433519] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:24:54 (1786353894) [ 1937.640572] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:24:57 (1786353897) [ 1940.671525] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:25:00 (1786353900) [ 1943.456066] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1673 sec ========== 05:25:03 (1786353903) [ 1943.588270] LNet: 144287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.593231] LNetError: 144287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.605649] LNet: Removed LNI 192.168.204.25@tcp [ 1943.903564] Key type .llcrypt unregistered [ 1943.905077] Key type ._llcrypt unregistered [ 1948.829876] Key type ._llcrypt registered [ 1948.831546] Key type .llcrypt registered [ 1949.040196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1949.047584] alg: No test for adler32 (adler32-zlib) [ 1949.925052] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1950.033389] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1951.631178] Key type lgssc registered [ 1952.103436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.538324] Lustre: Mounted lustre-client - version 2.17.56_50_g4393166 [ 1986.455654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1989.946145] Lustre: Unmounted lustre-client [ 2003.867980] Key type lgssc unregistered [ 2004.018818] LNet: 147236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.023970] LNetError: 147236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.034897] LNet: Removed LNI 192.168.204.25@tcp [ 2004.400315] Key type .llcrypt unregistered [ 2004.402440] Key type ._llcrypt unregistered