[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 433717515 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.004438] x2apic enabled [ 0.005015] Switched APIC routing to physical x2apic. [ 0.006021] kvm-guest: setup PV IPIs [ 0.010830] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012020] pid_max: default: 32768 minimum: 301 [ 0.013176] LSM: Security Framework initializing [ 0.014069] Yama: becoming mindful. [ 0.015136] SELinux: Initializing. [ 0.016190] *** VALIDATE selinux *** [ 0.023218] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032182] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033365] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034214] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037030] *** VALIDATE tmpfs *** [ 0.038479] *** VALIDATE proc *** [ 0.039315] *** VALIDATE cgroup *** [ 0.040019] *** VALIDATE cgroup2 *** [ 0.042170] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043199] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045045] Spectre V2 : User space: Vulnerable [ 0.046014] Speculative Store Bypass: Vulnerable [ 0.050067] debug: unmapping init [mem 0xffffffff85659000-0xffffffff85660fff] [ 0.052396] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053977] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055038] ... version: 2 [ 0.056016] ... bit width: 48 [ 0.057032] ... generic registers: 4 [ 0.058021] ... value mask: 0000ffffffffffff [ 0.059097] ... max period: 00007fffffffffff [ 0.060023] ... fixed-purpose events: 3 [ 0.061019] ... event mask: 000000070000000f [ 0.063286] rcu: Hierarchical SRCU implementation. [ 0.066188] smp: Bringing up secondary CPUs ... [ 0.067791] x86: Booting SMP configuration: [ 0.068044] .... node #0, CPUs: #1 #2 #3 [ 0.076499] smp: Brought up 1 node, 4 CPUs [ 0.078023] smpboot: Max logical packages: 1 [ 0.079015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122371] node 0 deferred pages initialised in 41ms [ 0.126169] devtmpfs: initialized [ 0.127296] x86/mm: Memory block size: 128MB [ 0.128197] gcov: version magic: 0x41383552 [ 0.131382] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137365] pinctrl core: initialized pinctrl subsystem [ 0.139279] [ 0.139875] ************************************************************* [ 0.142015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144015] ** ** [ 0.147015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149012] ** ** [ 0.151017] ** This means that this kernel is built to expose internal ** [ 0.154014] ** IOMMU data structures, which may compromise security on ** [ 0.156012] ** your system. ** [ 0.158021] ** ** [ 0.160015] ** If you see this message and you are not debugging the ** [ 0.164017] ** kernel, report this immediately to your vendor! ** [ 0.165000] ** ** [ 0.166016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169016] ************************************************************* [ 0.172865] NET: Registered protocol family 16 [ 0.174576] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183167] cpuidle: using governor menu [ 0.184913] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187894] PCI: Using configuration type 1 for base access [ 0.191231] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198491] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200446] cryptd: max_cpu_qlen set to 1000 [ 0.201320] ACPI: Added _OSI(Module Device) [ 0.203014] ACPI: Added _OSI(Processor Device) [ 0.204014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211337] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222585] ACPI: Interpreter enabled [ 0.223078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226016] ACPI: Using IOAPIC for interrupt routing [ 0.230191] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237659] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255442] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262152] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276231] acpiphp: Slot [2] registered [ 0.280363] acpiphp: Slot [5] registered [ 0.282225] acpiphp: Slot [6] registered [ 0.285257] acpiphp: Slot [3] registered [ 0.287360] acpiphp: Slot [4] registered [ 0.290213] acpiphp: Slot [7] registered [ 0.295243] acpiphp: Slot [8] registered [ 0.297330] acpiphp: Slot [9] registered [ 0.299153] acpiphp: Slot [10] registered [ 0.301288] acpiphp: Slot [11] registered [ 0.306236] acpiphp: Slot [12] registered [ 0.308114] acpiphp: Slot [13] registered [ 0.309195] acpiphp: Slot [14] registered [ 0.311106] acpiphp: Slot [15] registered [ 0.312383] acpiphp: Slot [16] registered [ 0.314118] acpiphp: Slot [17] registered [ 0.316155] acpiphp: Slot [18] registered [ 0.318155] acpiphp: Slot [19] registered [ 0.320228] acpiphp: Slot [20] registered [ 0.321244] acpiphp: Slot [21] registered [ 0.324148] acpiphp: Slot [22] registered [ 0.326193] acpiphp: Slot [23] registered [ 0.327111] acpiphp: Slot [24] registered [ 0.329154] acpiphp: Slot [25] registered [ 0.330094] acpiphp: Slot [26] registered [ 0.331198] acpiphp: Slot [27] registered [ 0.333118] acpiphp: Slot [28] registered [ 0.335110] acpiphp: Slot [29] registered [ 0.336108] acpiphp: Slot [30] registered [ 0.337164] acpiphp: Slot [31] registered [ 0.338076] PCI host bridge to bus 0000:00 [ 0.340021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356488] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360624] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366785] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.369038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378555] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380902] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.403018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408094] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416417] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463126] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.493023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.518317] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520441] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.522347] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.524512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.526368] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.532120] iommu: Default domain type: Passthrough [ 0.533360] SCSI subsystem initialized [ 0.534186] ACPI: bus type USB registered [ 0.536150] usbcore: registered new interface driver usbfs [ 0.537136] usbcore: registered new interface driver hub [ 0.539135] usbcore: registered new device driver usb [ 0.541236] pps_core: LinuxPPS API ver. 1 registered [ 0.543011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.545127] PTP clock support registered [ 0.547304] EDAC MC: Ver: 3.0.0 [ 0.548465] PCI: Using ACPI for IRQ routing [ 0.549000] NetLabel: Initializing [ 0.549000] NetLabel: domain hash size = 128 [ 0.549009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.550000] NetLabel: unlabeled traffic allowed by default [ 0.553341] vgaarb: loaded [ 0.554628] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.556009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.565159] clocksource: Switched to clocksource kvm-clock [ 0.679000] VFS: Disk quotas dquot_6.6.0 [ 0.680259] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.682285] *** VALIDATE ramfs *** [ 0.683233] *** VALIDATE hugetlbfs *** [ 0.684490] pnp: PnP ACPI init [ 0.686468] pnp: PnP ACPI: found 6 devices [ 0.700845] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.703545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.705265] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.706918] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.708931] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710670] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.712944] NET: Registered protocol family 2 [ 0.714883] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.719238] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.721746] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726676] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.729346] TCP: Hash tables configured (established 65536 bind 65536) [ 0.731925] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.734447] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.736689] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.738840] NET: Registered protocol family 1 [ 0.741632] RPC: Registered named UNIX socket transport module. [ 0.743283] RPC: Registered udp transport module. [ 0.744831] RPC: Registered tcp transport module. [ 0.746530] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.748390] NET: Registered protocol family 44 [ 0.749692] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.751369] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.752895] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.754820] PCI: CLS 0 bytes, default 64 [ 0.756092] Unpacking initramfs... [ 2.878624] debug: unmapping init [mem 0xffff8f353cc64000-0xffff8f353ffcffff] [ 2.891267] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.903016] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.910456] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.305046] Initialise system trusted keyrings [ 4.306744] Key type blacklist registered [ 4.312746] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.328603] zbud: loaded [ 4.339302] *** VALIDATE nfs *** [ 4.340410] *** VALIDATE nfs4 *** [ 4.345980] pstore: using deflate compression [ 4.367377] Platform Keyring initialized [ 4.644664] NET: Registered protocol family 38 [ 4.650275] Key type asymmetric registered [ 4.654646] Asymmetric key parser 'x509' registered [ 4.657481] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.660737] io scheduler mq-deadline registered [ 4.661889] io scheduler kyber registered [ 4.666321] io scheduler bfq registered [ 4.667839] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.670276] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.672595] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.674820] ACPI: Power Button [PWRF] [ 4.682763] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.689563] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.699349] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.730331] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.768940] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.774843] Non-volatile memory driver v1.3 [ 4.776470] Linux agpgart interface v0.103 [ 4.813420] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 4.816575] vda: detected capacity change from 0 to 74825728 [ 4.839721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.845136] vdb: detected capacity change from 0 to 1073741824 [ 4.864221] libphy: Fixed MDIO Bus: probed [ 4.871430] usbcore: registered new interface driver usbserial_generic [ 4.873954] usbserial: USB Serial support registered for generic [ 4.876273] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.881400] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.883851] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.886208] mousedev: PS/2 mouse device common for all mice [ 4.890279] rtc_cmos 00:05: RTC can wake from S4 [ 4.894456] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.894583] rtc_cmos 00:05: registered as rtc0 [ 4.901685] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.905985] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.909140] intel_pstate: CPU model not supported [ 4.913747] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.914267] hid: raw HID events driver (C) Jiri Kosina [ 4.919166] usbcore: registered new interface driver usbhid [ 4.920828] usbhid: USB HID core driver [ 4.922384] drop_monitor: Initializing network drop monitor service [ 4.924575] Initializing XFRM netlink socket [ 4.926299] NET: Registered protocol family 10 [ 4.928684] Segment Routing with IPv6 [ 4.931108] NET: Registered protocol family 17 [ 4.933955] mpls_gso: MPLS GSO support [ 4.941248] RAS: Correctable Errors collector initialized. [ 4.944876] AVX version of gcm_enc/dec engaged. [ 4.947240] AES CTR mode by8 optimization enabled [ 5.044813] sched_clock: Marking stable (5044732387, 0)->(6140073995, -1095341608) [ 5.050357] registered taskstats version 1 [ 5.052691] Loading compiled-in X.509 certificates [ 5.054988] zswap: loaded using pool lzo/zbud [ 5.085988] Key type big_key registered [ 5.099970] Key type encrypted registered [ 5.101551] ima: No TPM chip found, activating TPM-bypass! [ 5.103556] ima: Allocated hash algorithm: sha1 [ 5.105162] ima: No architecture policies found [ 5.106846] evm: Initialising EVM extended attributes: [ 5.109207] evm: security.selinux [ 5.110570] evm: security.ima [ 5.112113] evm: security.capability [ 5.113669] evm: HMAC attrs: 0x1 [ 5.117112] rtc_cmos 00:05: setting system clock to 2026-08-23 14:05:39 UTC (1787493939) [ 5.123944] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 5.127377] debug: unmapping init [mem 0xffffffff85382000-0xffffffff85658fff] [ 5.141103] Write protecting the kernel read-only data: 28672k [ 5.147401] debug: unmapping init [mem 0xffffffff83a03000-0xffffffff83bfffff] [ 5.152689] debug: unmapping init [mem 0xffffffff84314000-0xffffffff843fffff] [ 5.217248] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.237317] systemd[1]: Detected virtualization kvm. [ 5.240216] systemd[1]: Detected architecture x86-64. [ 5.246053] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.288820] systemd[1]: No hostname configured. [ 5.290936] systemd[1]: Set hostname to . [ 5.293489] random: systemd: uninitialized urandom read (16 bytes read) [ 5.296720] systemd[1]: Initializing machine ID from random generator. [ 5.651093] random: systemd: uninitialized urandom read (16 bytes read) [ 5.656058] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.700423] random: systemd: uninitialized urandom read (16 bytes read) [ 5.702838] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.711646] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.125604] device-mapper: uevent: version 1.0.3 [ 7.129740] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.513803] virtio_net virtio0 ens2: renamed from eth0 [ 8.896146] scsi host0: ata_piix [ 8.975497] scsi host1: ata_piix [ 8.978039] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.984350] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.631311] random: crng init done [ 13.632714] random: 7 urandom warning(s) missed due to ratelimiting [ 14.559691] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.559203] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.994489] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.851350] SELinux: Disabled at runtime. [ 20.001093] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.019278] systemd[1]: Detected virtualization kvm. [ 20.021116] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.826663] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.836246] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.862991] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.878981] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.896974] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.927940] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.945944] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 22.120948] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.594696] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.892628] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.028757] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.156326] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.254995] EDAC sbridge: Ver: 1.1.2 [ 29.238904] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.054761] NFS: Registering the id_resolver key type [ 30.062591] Key type id_resolver registered [ 30.066138] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg144-client login: [ 94.216135] libcfs: loading out-of-tree module taints kernel. [ 94.337035] Key type ._llcrypt registered [ 94.342956] Key type .llcrypt registered [ 95.278658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.303849] alg: No test for adler32 (adler32-zlib) [ 97.078176] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 98.260845] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 100.017292] Key type lgssc registered [ 101.885383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 251.021399] Lustre: Mounted lustre-client - version 2.17.57_80_gd2e8623 [ 256.847494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.267580] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing check_logdir /tmp/testlogs/ [ 276.447248] Lustre: lustre-OST0000-osc-ffff8f3589ce9800: disconnect after 23s idle [ 277.468378] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing yml_node [ 280.868906] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 283.708063] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 286.077561] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 287.438575] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Aug 23 10:10:20 EDT 2026 [ 291.529505] hrtimer: interrupt took 1635389 ns [ 304.301960] Lustre: Unmounted lustre-client [ 333.172830] Key type lgssc unregistered [ 333.565619] LNet: 8433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.581583] LNetError: 8433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.599414] LNet: Removed LNI 192.168.201.44@tcp [ 334.445124] Key type .llcrypt unregistered [ 334.448725] Key type ._llcrypt unregistered [ 363.897886] Key type ._llcrypt registered [ 363.900605] Key type .llcrypt registered [ 364.131404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 364.166172] alg: No test for adler32 (adler32-zlib) [ 365.110246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 365.202269] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 365.231326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 369.055026] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 369.278251] LNet: 9321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.281928] LNetError: 9321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.306790] LNet: Removed LNI 192.168.201.44@tcp [ 370.356360] Key type .llcrypt unregistered [ 370.357982] Key type ._llcrypt unregistered [ 377.125915] Key type ._llcrypt registered [ 377.132277] Key type .llcrypt registered [ 378.656242] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:11:51 (1787494311) [ 378.911150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 378.926136] alg: No test for adler32 (adler32-zlib) [ 379.912889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.035432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.056279] LNet: 9742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.073109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.188208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.204473] LNet: 9749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 388.616931] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:12:00 (1787494320) [ 389.010472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.044797] LNet: 10327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.062705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.150526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 395.791325] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:12:08 (1787494328) [ 395.920578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.974482] LNet: 10906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.012880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.138608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 403.613387] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:12:16 (1787494336) [ 403.753392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.781757] LNet: 11496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.818839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.884881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 410.601526] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:12:23 (1787494343) [ 410.722051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.749852] LNet: 12075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.779574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.898360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 411.537077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 418.458691] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:12:31 (1787494351) [ 418.522941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.565164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.617530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 418.854737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 419.113792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 424.463976] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:12:37 (1787494357) [ 424.679331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.704171] LNet: 13268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.711608] LNet: 13268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 424.739505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.851065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 430.682197] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:12:43 (1787494363) [ 430.771924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.828210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.908073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 430.938764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.464794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 431.504832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 431.857861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 431.890719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.218403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 432.256967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 432.568752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 432.605499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.916585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 432.946422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.295994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 433.334518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.706179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 433.751513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.197609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 434.226276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.620978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 434.689121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.035949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 435.081441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.463982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 435.497676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.841948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.871942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 441.510665] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:12:54 (1787494374) [ 441.591732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.628824] LNet: 14528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.633363] LNet: 14528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 441.666910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.774955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 441.810985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 448.612261] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:13:01 (1787494381) [ 448.729978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.782538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.907647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 448.956927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.757292] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:13:08 (1787494388) [ 456.177576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.239209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.328733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 456.370822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 463.006111] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:13:15 (1787494395) [ 463.092376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.158993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.315039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 463.366586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 471.112070] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:13:23 (1787494403) [ 471.246070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.305874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.384774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 471.414375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 478.868723] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:13:31 (1787494411) [ 479.045531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.072667] LNet: 17465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.081881] LNet: 17465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 479.115562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.229299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 479.268310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.495903] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:13:38 (1787494418) [ 485.637492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.698373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.826933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.362653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.829426] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:13:45 (1787494425) [ 493.203509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.261964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.545677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.779529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.516492] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:13:53 (1787494433) [ 500.618927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.687114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.861597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.151400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.452439] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:14:00 (1787494440) [ 507.708724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.757420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.192356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.433756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.385655] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:14:08 (1787494448) [ 515.643971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.676511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.735930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.835781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.122452] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:14:15 (1787494455) [ 522.293419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.410764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.512660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 522.622435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.911355] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:14:22 (1787494462) [ 529.985495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.041250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.146188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 536.154822] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:14:29 (1787494469) [ 536.262377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.334443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.412276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 542.468424] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:14:35 (1787494475) [ 542.668086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.717387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.772854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 543.018890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 548.991146] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:14:42 (1787494482) [ 549.137704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.170858] LNet: 23494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.176868] LNet: 23494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 549.246376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.325782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 549.365752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 555.895052] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:14:49 (1787494489) [ 556.239705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.309906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.375898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 556.411210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 562.867397] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:14:55 (1787494495) [ 562.950499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.996120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.065242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 563.100115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 569.540336] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:15:02 (1787494502) [ 569.950457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.003955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.110438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 570.179676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 576.070245] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:15:09 (1787494509) [ 576.165433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.231496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.331456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 576.387481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 576.412146] LNetError: 25851:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 576.487682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 576.531721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 582.699131] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:15:15 (1787494515) [ 582.791802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.864214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.007507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GYga4TnMdq [ 583.182888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GYga4TnMdq [ 583.320811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GYga4TnMdq [ 590.360765] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:15:23 (1787494523) [ 590.699439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.756644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.814843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 590.871342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 590.939875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 598.673860] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:15:31 (1787494531) [ 598.780320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.830960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.920783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 598.977237] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 600.220145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 601.836262] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 603.831228] Key type lgssc registered [ 604.783822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 608.289366] LNetError: 27610:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.44@tcp99 has no available nets [ 611.185981] Key type lgssc unregistered [ 611.578076] LNetError: 28722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.603261] LNet: Removed LNI 192.168.201.44@tcp [ 612.263828] Key type .llcrypt unregistered [ 612.265818] Key type ._llcrypt unregistered [ 619.512840] Key type ._llcrypt registered [ 619.515324] Key type .llcrypt registered [ 621.010868] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:15:53 (1787494553) [ 621.683313] Key type .llcrypt unregistered [ 621.686259] Key type ._llcrypt unregistered [ 630.181899] Key type ._llcrypt registered [ 630.196909] Key type .llcrypt registered [ 630.680475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 630.742129] alg: No test for adler32 (adler32-zlib) [ 631.751880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.861611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 631.930806] LNet: 29643:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 631.942089] LNet: 29643:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 631.951623] LNet: 29643:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 631.983110] LNet: Message forwarding will be enabled in 2s [ 635.104301] LNet: Message forwarding enabled [ 635.564139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 635.581141] LNet: Message forwarding will stop in 2s to 6s [ 639.212939] LNet: Message forwarding disabled [ 640.073239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 640.115090] LNet: 29741:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 640.128173] LNet: 29741:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 640.143938] LNet: 29741:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 640.160683] LNet: Message forwarding will be enabled in 2s [ 640.384588] LNet: 29754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.014625] Key type .llcrypt unregistered [ 641.015911] Key type ._llcrypt unregistered [ 654.630941] Key type ._llcrypt registered [ 654.634694] Key type .llcrypt registered [ 656.444457] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:16:28 (1787494588) [ 656.811064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.826293] alg: No test for adler32 (adler32-zlib) [ 657.794772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.912810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 665.566688] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:16:38 (1787494598) [ 665.664355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.700369] LNet: 31152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.745098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.829613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 665.882446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 665.916152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 665.948399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 665.974033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 666.006249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 666.037637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 666.083061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 666.155597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 666.231470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 666.271596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 666.332879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 666.370767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 666.401982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 666.446640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 666.491209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 666.544179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 666.576379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 666.616975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 666.645691] LNetError: 31204:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 673.600218] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:16:46 (1787494606) [ 673.943858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.964348] LNet: 31775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.993709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.093854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 681.127456] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:16:54 (1787494614) [ 682.546519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg144-server [ 684.460646] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:16:57 (1787494617) [ 684.542297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.568681] LNet: 32700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.603412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.713555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.761635] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 685.611663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[45-49]@tcp [ 685.651854] LNet: 32841:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 685.697141] LNetError: 32708:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 685.711934] LNetError: 32708:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.45@tcp has gone from up to down [ 695.158775] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:17:07 (1787494627) [ 697.677097] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg144-server [ 699.979488] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:17:12 (1787494632) [ 700.184960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.244430] LNet: 33773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.251899] LNetError: 33773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.294987] LNet: Removed LNI 192.168.201.44@tcp [ 700.384591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.529468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.589804] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 701.848134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[45-50/2]@tcp [ 701.905791] LNet: 33895:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 701.923977] LNet: 33895:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 701.964213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[45-50/2]@tcp [ 701.978540] LNetError: 33782:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 701.986739] LNetError: 33782:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 701.992642] LNetError: 33782:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.47@tcp has gone from up to down [ 701.997882] LNetError: 33782:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 709.424343] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:17:22 (1787494642) [ 709.827194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.853330] LNet: 34487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.866398] LNetError: 34487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.884922] LNet: Removed LNI 192.168.201.44@tcp [ 709.900222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.002614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 710.047212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.072199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.136282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.174502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.210754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.274838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 710.335880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.391635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.457819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.522597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.558626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.608310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 710.682619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.723576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.785598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.895734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.933366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.002158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 711.095297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.130515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.188347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.275493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.311256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.349119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 711.404824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.424897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.459272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.552088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.580555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.621192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 711.675275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.719218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.779191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.718989] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:17:30 (1787494650) [ 719.451785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg144-server [ 721.568513] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:17:34 (1787494654) [ 723.326295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg144-server [ 724.687806] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:17:37 (1787494657) [ 724.875907] LNet: 35887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.883161] LNet: 35887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 725.665349] Key type .llcrypt unregistered [ 725.667782] Key type ._llcrypt unregistered [ 732.382348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.546455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.549734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.551358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.781206] Key type ._llcrypt registered [ 735.782944] Key type .llcrypt registered [ 736.032392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.073027] alg: No test for adler32 (adler32-zlib) [ 737.155892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.265452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.340038] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 738.327656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 738.366906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.739919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 741.774825] LNet: Removed LNI 10.1.2.3@tcp [ 742.011312] LNet: 36607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.031206] LNetError: 36607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.041894] LNet: Removed LNI 192.168.201.44@tcp [ 743.216826] Key type .llcrypt unregistered [ 743.226827] Key type ._llcrypt unregistered [ 762.094402] Key type ._llcrypt registered [ 762.101517] Key type .llcrypt registered [ 763.569476] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:18:16 (1787494696) [ 764.094347] Key type .llcrypt unregistered [ 764.095946] Key type ._llcrypt unregistered [ 770.660240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 770.818740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 770.836156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 770.846923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 773.635467] Key type ._llcrypt registered [ 773.639220] Key type .llcrypt registered [ 773.995551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.038512] alg: No test for adler32 (adler32-zlib) [ 775.157544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.192679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.207545] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 775.777209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.796283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.113521] LNet: 38110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.118962] LNetError: 38110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 778.125674] LNet: Removed LNI 10.1.2.3@tcp [ 778.136859] LNet: Removed LNI 192.168.201.44@tcp [ 779.303979] Key type .llcrypt unregistered [ 779.314376] Key type ._llcrypt unregistered [ 792.433215] Key type ._llcrypt registered [ 792.435321] Key type .llcrypt registered [ 793.726340] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:18:46 (1787494726) [ 794.252571] Key type .llcrypt unregistered [ 794.258312] Key type ._llcrypt unregistered [ 799.928658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.065201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.071957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.090085] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 802.915571] Key type ._llcrypt registered [ 802.918142] Key type .llcrypt registered [ 803.085714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.100207] alg: No test for adler32 (adler32-zlib) [ 804.034619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.113148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 804.145419] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 806.145970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.170614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.918238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 807.953148] LNet: Removed LNI 10.1.2.3@tcp [ 807.973757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 808.027820] LNet: Removed LNI 10.1.2.31@tcp [ 808.043377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 808.059528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.862486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 808.879559] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 809.688972] LNet: 40037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.698220] LNetError: 40037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 809.708677] LNet: Removed LNI 10.1.2.31@tcp [ 809.723219] LNet: Removed LNI 10.1.2.3@tcp [ 810.282409] Key type .llcrypt unregistered [ 810.284994] Key type ._llcrypt unregistered [ 821.680486] Key type ._llcrypt registered [ 821.681861] Key type .llcrypt registered [ 822.786850] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:19:15 (1787494755) [ 823.167728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.182040] alg: No test for adler32 (adler32-zlib) [ 824.086961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.127092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.143403] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 824.893635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.926230] LNet: 40993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.947495] LNetError: 40993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.975403] LNet: Removed LNI 192.168.201.44@tcp [ 824.989639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.024356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.44@tcp [ 825.047502] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 825.763342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.44@tcp [ 825.832340] LNet: Removed LNI 192.168.201.44@tcp [ 825.912479] LNet: 41129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.570511] Key type .llcrypt unregistered [ 826.572247] Key type ._llcrypt unregistered [ 836.605557] Key type ._llcrypt registered [ 836.606705] Key type .llcrypt registered [ 838.224967] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:19:30 (1787494770) [ 840.772896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg144-server [ 842.771477] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:19:35 (1787494775) [ 843.618312] Key type .llcrypt unregistered [ 843.620521] Key type ._llcrypt unregistered [ 850.255799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.296742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.299194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.303103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.930719] Key type ._llcrypt registered [ 852.934561] Key type .llcrypt registered [ 853.155941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.166314] alg: No test for adler32 (adler32-zlib) [ 854.119881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.252680] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 855.006641] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 856.890978] LNet: 42972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.898565] LNetError: 42972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.903536] LNet: Removed LNI 192.168.201.44@tcp1 [ 856.912881] LNet: Removed LNI 10.1.2.3@tcp2 [ 857.415397] Key type .llcrypt unregistered [ 857.417152] Key type ._llcrypt unregistered [ 868.535775] Key type ._llcrypt registered [ 868.537450] Key type .llcrypt registered [ 869.643408] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:20:02 (1787494802) [ 869.978925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.993700] alg: No test for adler32 (adler32-zlib) [ 870.991519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.075738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.102974] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 871.855784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.884295] LNet: 43945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.900071] LNetError: 43945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.917815] LNet: Removed LNI 192.168.201.44@tcp [ 871.934780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.038343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 872.070741] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 878.924562] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:20:12 (1787494812) [ 879.128571] LNet: 44537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.133808] LNetError: 44537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.145386] LNet: Removed LNI 192.168.201.44@tcp [ 879.716818] Key type .llcrypt unregistered [ 879.720802] Key type ._llcrypt unregistered [ 885.605688] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.958183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.970583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.980147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.681556] Key type ._llcrypt registered [ 888.688274] Key type .llcrypt registered [ 888.837843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.853264] alg: No test for adler32 (adler32-zlib) [ 889.803503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.882120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.925642] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 890.591186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 890.606828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.351659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.368967] LNet: 45281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.374503] LNetError: 45281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.379384] LNet: Removed LNI 10.1.2.3@tcp [ 892.387560] LNet: Removed LNI 192.168.201.44@tcp [ 892.401777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.473860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.499654] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 892.505045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.202464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.243512] LNet: 45309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.251477] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.270983] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 893.278789] LNet: Removed LNI 10.1.2.3@tcp [ 893.295824] LNet: Removed LNI 192.168.201.44@tcp [ 893.344180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.476994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.527642] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 893.540878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.186159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 894.222755] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 895.111662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.126343] LNet: 45471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.131292] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.136348] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 895.144642] LNet: Removed LNI 10.1.2.3@tcp [ 895.148783] LNet: Removed LNI 192.168.201.44@tcp [ 895.164312] LNet: Removed LNI 192.168.201.44@tcp2 [ 895.212634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.330408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 895.359887] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 895.365877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.373284] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 903.120629] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:20:36 (1787494836) [ 903.256707] LNet: 46064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.263965] LNetError: 46064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.270853] LNetError: 46064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 903.290087] LNet: Removed LNI 10.1.2.3@tcp [ 903.291757] LNet: Removed LNI 192.168.201.44@tcp [ 903.334342] LNet: Removed LNI 192.168.201.44@tcp2 [ 904.175950] Key type .llcrypt unregistered [ 904.177313] Key type ._llcrypt unregistered [ 910.185234] Key type ._llcrypt registered [ 910.193135] Key type .llcrypt registered [ 910.394636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.405488] alg: No test for adler32 (adler32-zlib) [ 911.339150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.393454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.417790] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 912.072695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.089983] LNet: 46470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.097594] LNetError: 46470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.114811] LNet: Removed LNI 192.168.201.44@tcp [ 912.132994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.185371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.206541] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 912.602273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.619387] LNet: 46500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.626571] LNetError: 46500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.645361] LNet: Removed LNI 192.168.201.44@tcp [ 912.709286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.826100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.894891] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 913.498541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.528473] LNet: Removed LNI 192.168.201.44@tcp [ 913.565751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.674763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.712376] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 914.313098] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.553530] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.562371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.567455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.893281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.914475] LNet: 46598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.920170] LNet: 46598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 915.928176] LNetError: 46598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.932161] LNetError: 46598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 915.951637] LNet: Removed LNI 192.168.201.44@tcp [ 915.977175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.068533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.102732] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 923.216667] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:20:56 (1787494856) [ 923.431217] LNet: 47195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.439830] LNetError: 47195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.452264] LNet: Removed LNI 192.168.201.44@tcp [ 923.965880] Key type .llcrypt unregistered [ 923.967611] Key type ._llcrypt unregistered [ 929.439966] Key type ._llcrypt registered [ 929.441774] Key type .llcrypt registered [ 929.614417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.625653] alg: No test for adler32 (adler32-zlib) [ 930.502470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.546476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.573511] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 931.350314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.372136] LNet: 47598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.386677] LNetError: 47598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.416187] LNet: Removed LNI 192.168.201.44@tcp [ 931.439383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.491790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 931.505983] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 936.947775] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:21:10 (1787494870) [ 937.149415] LNet: 48190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.153919] LNetError: 48190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.170201] LNet: Removed LNI 192.168.201.44@tcp [ 937.614681] Key type .llcrypt unregistered [ 937.616354] Key type ._llcrypt unregistered [ 942.837768] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.994339] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.002254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.009822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.554939] Key type ._llcrypt registered [ 945.558905] Key type .llcrypt registered [ 945.746655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.754977] alg: No test for adler32 (adler32-zlib) [ 946.646469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.709574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.758694] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 947.513572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.532850] LNet: 48645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.538721] LNetError: 48645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.561542] LNet: Removed LNI 192.168.201.44@tcp [ 947.594498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.682540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 947.715500] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 953.605172] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:21:26 (1787494886) [ 953.801576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.823103] LNet: 49244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.829656] LNetError: 49244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.847482] LNet: Removed LNI 192.168.201.44@tcp [ 953.869957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.943796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 959.385906] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:21:32 (1787494892) [ 959.488771] LNet: 49818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.042203] Key type .llcrypt unregistered [ 960.046851] Key type ._llcrypt unregistered [ 966.251689] Key type ._llcrypt registered [ 966.253357] Key type .llcrypt registered [ 966.555915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.573509] alg: No test for adler32 (adler32-zlib) [ 967.514410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.587155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.610450] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 968.465328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.482708] LNet: 50228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.488339] LNetError: 50228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.504168] LNet: Removed LNI 192.168.201.44@tcp [ 968.523867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.581364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 968.617359] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 975.158457] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:21:48 (1787494908) [ 975.258364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.277372] LNet: 50816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.287896] LNetError: 50816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.312886] LNet: Removed LNI 192.168.201.44@tcp [ 975.340121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.379854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 981.690517] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:21:54 (1787494914) [ 983.698036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 985.430220] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:21:58 (1787494918) [ 987.082865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 988.547422] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:22:01 (1787494921) [ 990.231623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 992.466965] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:22:05 (1787494925) [ 994.905651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 997.022579] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:22:09 (1787494929) [ 998.602710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1000.168484] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:22:13 (1787494933) [ 1000.610370] LNet: 52876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.442274] Key type .llcrypt unregistered [ 1001.444101] Key type ._llcrypt unregistered [ 1008.140383] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1008.473376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1008.487423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1008.500591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1011.162918] Key type ._llcrypt registered [ 1011.165493] Key type .llcrypt registered [ 1011.323726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.337754] alg: No test for adler32 (adler32-zlib) [ 1012.270222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.375280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1012.399296] LNet: Added LNI 192.168.201.44@tcp [8/512/0/180] [ 1012.405737] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1012.408038] LNet: 53199:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1012.541126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.572923] LNet: 53218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.580600] LNetError: 53218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.595291] LNet: Removed LNI 10.1.2.3@tcp [ 1012.601709] LNetError: 53218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.633089] LNet: Removed LNI 192.168.201.44@tcp [ 1012.649770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.704479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1012.730716] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1012.791790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.806839] LNet: 53242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.812587] LNetError: 53242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.823958] LNet: Removed LNI 192.168.201.44@tcp [ 1012.850916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.957942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.003858] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1013.012412] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1013.015582] LNet: 53250:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1013.133517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.163617] LNet: 53268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.170784] LNetError: 53268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.174510] LNet: Removed LNI 10.1.2.3@tcp [ 1013.177090] LNetError: 53268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.198817] LNet: Removed LNI 192.168.201.44@tcp [ 1013.245248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.335527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.381453] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1013.393386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1013.398333] LNet: 53276:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1013.683855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.727979] LNet: 53302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.742930] LNetError: 53302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.769195] LNet: Removed LNI 10.1.2.3@tcp [ 1013.772812] LNetError: 53302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.792172] LNet: Removed LNI 192.168.201.44@tcp [ 1013.827788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.894178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.931839] LNet: Added LNI 192.168.201.44@tcp [8/256/0/182] [ 1013.937237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1014.073929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.121148] LNet: 53330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.129799] LNetError: 53330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.142406] LNet: Removed LNI 10.1.2.3@tcp [ 1014.147155] LNetError: 53330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.159306] LNet: Removed LNI 192.168.201.44@tcp [ 1014.184362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.260196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.309208] LNet: Added LNI 192.168.201.44@tcp [8/256/0/182] [ 1014.321754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1014.323675] LNet: 53338:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.437805] LNet: Removed LNI 10.1.2.3@tcp [ 1014.450365] LNet: Removed LNI 192.168.201.44@tcp [ 1015.260908] Key type .llcrypt unregistered [ 1015.268341] Key type ._llcrypt unregistered [ 1027.480191] Key type ._llcrypt registered [ 1027.482776] Key type .llcrypt registered [ 1028.761367] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:22:41 (1787494961) [ 1030.568217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1032.372379] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:22:45 (1787494965) [ 1032.739508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.758578] alg: No test for adler32 (adler32-zlib) [ 1033.757931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.816917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1033.854471] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1035.205906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.44 [ 1035.238309] LNet: 54605:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1035.244752] LNet: Removed LNI 192.168.201.44@tcp [ 1042.077132] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:22:54 (1787494974) [ 1042.196702] LNet: 55170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.205503] LNetError: 55170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.236353] LNet: Removed LNI 192.168.201.44@tcp [ 1042.915193] Key type .llcrypt unregistered [ 1042.923363] Key type ._llcrypt unregistered [ 1051.311853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1051.524127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1053.537601] Key type ._llcrypt registered [ 1053.538832] Key type .llcrypt registered [ 1053.961306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.005178] alg: No test for adler32 (adler32-zlib) [ 1055.531403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.733833] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:23:16 (1787494996) [ 1064.083422] LNet: 56088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.091892] LNetError: 56088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.133228] LNet: Removed LNI 10.1.2.3@tcp [ 1064.760610] Key type .llcrypt unregistered [ 1064.762847] Key type ._llcrypt unregistered [ 1071.256621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1071.414564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1071.417992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1071.426878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1073.226681] Key type ._llcrypt registered [ 1073.229886] Key type .llcrypt registered [ 1073.414617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.444469] alg: No test for adler32 (adler32-zlib) [ 1074.423323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1082.198316] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:23:34 (1787495014) [ 1083.593543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1085.371742] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:23:38 (1787495018) [ 1085.516131] LNet: 57264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.528223] LNetError: 57264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1085.550229] LNet: Removed LNI 10.1.2.3@tcp [ 1086.316374] Key type .llcrypt unregistered [ 1086.319882] Key type ._llcrypt unregistered [ 1094.378181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1094.695163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1094.699524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1094.703635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1097.599485] Key type ._llcrypt registered [ 1097.601113] Key type .llcrypt registered [ 1097.838365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.879817] alg: No test for adler32 (adler32-zlib) [ 1099.003566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.095237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1109.005876] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:24:02 (1787495042) [ 1109.704793] LNet: 58506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.727711] LNetError: 58506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1109.745931] LNet: Removed LNI 10.1.2.3@tcp [ 1110.804381] Key type .llcrypt unregistered [ 1110.806608] Key type ._llcrypt unregistered [ 1118.628225] Key type ._llcrypt registered [ 1118.629478] Key type .llcrypt registered [ 1118.899233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.917836] alg: No test for adler32 (adler32-zlib) [ 1119.855622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1119.897208] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1129.448948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1130.311582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_interrupt -r 1 [ 1130.361120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_interrupt -r 1 [ 1130.395928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1130.420387] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1130.720040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.776223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.827205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.879754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.937476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.009422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.079073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.109585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.141283] LNet: 58948:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.167494] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1131.182823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.185451] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1131.221971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.256326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.314962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.346139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.405744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.461293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.148951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_dropped -r 1 [ 1132.243884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_dropped -r 1 [ 1132.339650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1132.374310] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1132.702549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.796414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.833213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.879935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.932946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.002437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.092861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.182761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.204958] LNet: 59058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.225356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.251528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.297114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.364987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.444463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.487829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.548474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.186897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_aborted -r 1 [ 1134.251867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_aborted -r 1 [ 1134.326353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1134.377415] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1134.732833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.794784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.857338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.936370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.013310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.064402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.117440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.174509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.204766] LNet: 59167:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.283309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.342524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.387537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.428699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.484619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.563988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.644354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.496055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_no_route -r 1 [ 1136.566395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_no_route -r 1 [ 1136.617862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1136.908214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.944205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.011421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.058707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.103251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.159629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.226609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.292990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.334608] LNet: 59276:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.362450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.446987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.521107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.575887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.658552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.741888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.798084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.519473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_timeout -r 1 [ 1138.574430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_timeout -r 1 [ 1138.629049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1138.679609] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1138.685824] LNet: Skipped 1 previous similar message [ 1139.000145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.050531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.102089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.148777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.193677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.233634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.274556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.312763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.349611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.378383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.414410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.488589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.525071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.566712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.605525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.344751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_error -r 1 [ 1140.391316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_error -r 1 [ 1140.444383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1140.821231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.867493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.927548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.967399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.009650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.065449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.109691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.139168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.169540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.205063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.238422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.293745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.327820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.371795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.418321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.853895] LNet: 59540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.864897] LNetError: 59540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.916933] LNet: Removed LNI 192.168.201.44@tcp [ 1142.526197] Key type .llcrypt unregistered [ 1142.528430] Key type ._llcrypt unregistered [ 1162.178614] Key type ._llcrypt registered [ 1162.180678] Key type .llcrypt registered [ 1163.587735] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:24:56 (1787495096) [ 1164.652191] Key type .llcrypt unregistered [ 1164.656400] Key type ._llcrypt unregistered [ 1171.265412] Key type ._llcrypt registered [ 1171.271439] Key type .llcrypt registered [ 1171.492945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.514655] alg: No test for adler32 (adler32-zlib) [ 1172.569538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.641628] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1184.653857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1196.059926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1196.142648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1196.171374] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1198.152133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_interrupt -r 1 [ 1198.223595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_interrupt -r 1 [ 1198.328830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_interrupt -r 1 [ 1198.440452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_interrupt -r 1 [ 1198.561852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_interrupt -r 1 [ 1198.634213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_interrupt -r 1 [ 1198.689783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_interrupt -r 1 [ 1198.762491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_interrupt -r 1 [ 1198.846880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1198.872558] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1198.885337] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1199.288402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.402753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.474631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.506247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.569785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.619812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.695673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.774683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.863735] LNet: 61014:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.907035] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1199.918575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.925275] LNet: Skipped 3 previous similar messages [ 1199.987708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.061855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.118723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.165305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.235667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.307037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.207377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_dropped -r 1 [ 1201.276109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_dropped -r 1 [ 1201.352300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_dropped -r 1 [ 1201.466628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_dropped -r 1 [ 1201.557075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_dropped -r 1 [ 1201.639140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_dropped -r 1 [ 1201.741117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_dropped -r 1 [ 1201.814429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_dropped -r 1 [ 1201.882855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1201.915202] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1202.261551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.323425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.374502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.433048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.475134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.535425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.583203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.618205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.635324] LNet: 61136:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.653881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.692237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.712946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.759247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.809240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.847547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.901022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.722474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_aborted -r 1 [ 1203.758767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_aborted -r 1 [ 1203.792281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_aborted -r 1 [ 1203.827483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_aborted -r 1 [ 1203.879079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_aborted -r 1 [ 1203.927756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_aborted -r 1 [ 1203.975480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_aborted -r 1 [ 1204.040232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_aborted -r 1 [ 1204.091103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1204.127033] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1204.139763] LNet: Skipped 2 previous similar messages [ 1204.449857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.488876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.538578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.603360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.682515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.733749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.789995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.866678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.892529] LNet: 61257:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.928417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.978541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.055975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.145900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.220902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.244257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.291288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1206.068647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_no_route -r 1 [ 1206.124243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_no_route -r 1 [ 1206.207477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_no_route -r 1 [ 1206.253741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_no_route -r 1 [ 1206.304564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_no_route -r 1 [ 1206.338969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_no_route -r 1 [ 1206.372869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_no_route -r 1 [ 1206.410506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_no_route -r 1 [ 1206.437140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1206.804877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.884796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.967872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.041528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.101425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.167642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.206665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.227328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.239745] LNet: 61378:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.253513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.286261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.357171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.437811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.501234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.587645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.717977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.888819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_timeout -r 1 [ 1208.968539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_timeout -r 1 [ 1209.018265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_timeout -r 1 [ 1209.061683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_timeout -r 1 [ 1209.110488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_timeout -r 1 [ 1209.148946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_timeout -r 1 [ 1209.191645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_timeout -r 1 [ 1209.265580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_timeout -r 1 [ 1209.308795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1209.329974] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1209.334556] LNet: Skipped 5 previous similar messages [ 1209.545930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.580911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.621523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.647619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.712831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.766500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.863491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.912981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.972793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.035966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.106646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.172575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.239978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.326554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.366830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.953338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_error -r 1 [ 1210.992648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_error -r 1 [ 1211.027376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_error -r 1 [ 1211.065495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_error -r 1 [ 1211.120748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_error -r 1 [ 1211.172744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_error -r 1 [ 1211.220048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_error -r 1 [ 1211.264906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_error -r 1 [ 1211.322330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1211.587499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.617907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.646141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.677996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.702222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.753361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.786116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.819228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.844698] LNet: 61621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.853649] LNet: 61621:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1211.871861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.907892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.932934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.969259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.013839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.049211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.107664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.872584] LNet: 61701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.881559] LNetError: 61701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.911369] LNet: Removed LNI 192.168.201.44@tcp [ 1218.913286] LNetError: 61701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.958814] LNet: Removed LNI 192.168.201.44@tcp1 [ 1219.602157] Key type .llcrypt unregistered [ 1219.608706] Key type ._llcrypt unregistered [ 1239.804163] Key type ._llcrypt registered [ 1239.805714] Key type .llcrypt registered [ 1241.365993] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:26:13 (1787495173) [ 1241.998384] Key type .llcrypt unregistered [ 1242.000052] Key type ._llcrypt unregistered [ 1249.175826] Key type ._llcrypt registered [ 1249.182842] Key type .llcrypt registered [ 1249.405683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.418222] alg: No test for adler32 (adler32-zlib) [ 1250.357739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.393774] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1258.115150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1258.827694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_dropped -r 1 [ 1258.886630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_dropped -r 1 [ 1258.923495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1258.948231] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1259.181798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.214296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.256377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.298718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.337916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.358840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.416482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.483190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.515412] LNet: 62969:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.551097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.609636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.646293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.706168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.738930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.787405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.822647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.271535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_error -r 1 [ 1260.338651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_error -r 1 [ 1260.392323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1260.412154] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1260.602213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.630201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.659269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.709075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.760413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.800079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.837635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.872651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.891439] LNet: 63078:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.913660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.936489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.960374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.988571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.016955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.046518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.083245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.640867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_timeout -r 1 [ 1261.680140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_timeout -r 1 [ 1261.723925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1261.749165] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1262.029321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.060750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.089913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.139716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.188963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.250620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.297898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.341167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.365481] LNet: 63188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.399875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.449871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.503490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.573654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.613195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.680970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.750574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1263.098374] LNet: 63233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.109417] LNetError: 63233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.154419] LNet: Removed LNI 192.168.201.44@tcp [ 1263.547702] Key type .llcrypt unregistered [ 1263.550386] Key type ._llcrypt unregistered [ 1282.361245] Key type ._llcrypt registered [ 1282.366901] Key type .llcrypt registered [ 1283.816460] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:26:56 (1787495216) [ 1284.750462] Key type .llcrypt unregistered [ 1284.752681] Key type ._llcrypt unregistered [ 1292.272820] Key type ._llcrypt registered [ 1292.275863] Key type .llcrypt registered [ 1292.538339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.548398] alg: No test for adler32 (adler32-zlib) [ 1293.498257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.539423] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1302.212086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1312.643427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.697803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.743820] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1314.448939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_dropped -r 1 [ 1314.494740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e remote_dropped -r 1 [ 1314.525634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_dropped -r 1 [ 1314.569429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e remote_dropped -r 1 [ 1314.618748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e remote_dropped -r 1 [ 1314.671990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e remote_dropped -r 1 [ 1314.709770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e remote_dropped -r 1 [ 1314.750335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e remote_dropped -r 1 [ 1314.789546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1314.813651] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1314.818901] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.144@tcp [ 1315.025208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1315.060265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1315.083022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.111506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.144150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.177258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.210980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.245172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.266498] LNet: 64703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.293491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.333310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.375147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.412309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.501572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.561489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.588366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.608646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.281623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_error -r 1 [ 1316.308813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e remote_error -r 1 [ 1316.342952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_error -r 1 [ 1316.386559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e remote_error -r 1 [ 1316.425590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e remote_error -r 1 [ 1316.491348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e remote_error -r 1 [ 1316.550893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e remote_error -r 1 [ 1316.637377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e remote_error -r 1 [ 1316.697205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1316.729507] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1316.737163] LNet: Skipped 2 previous similar messages [ 1317.034704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.067361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.106157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1317.140205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1317.194472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1317.244572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1317.286762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1317.327200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1317.353893] LNet: 64827:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.372198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1317.414801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1317.442937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.483114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1317.531419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1317.569421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1317.614295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.654427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.326063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_timeout -r 1 [ 1318.362735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e remote_timeout -r 1 [ 1318.415950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_timeout -r 1 [ 1318.446337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e remote_timeout -r 1 [ 1318.494183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e remote_timeout -r 1 [ 1318.538232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e remote_timeout -r 1 [ 1318.588214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e remote_timeout -r 1 [ 1318.670550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e remote_timeout -r 1 [ 1318.744389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.144@tcp [ 1318.778804] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1319.214886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.261808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1319.315794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1319.382310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1319.442607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1319.490935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1319.543283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1319.569166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1319.580477] LNet: 64950:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1319.627350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1319.676309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1319.712679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1319.736332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1319.765453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.793257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1319.824536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.860543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1326.097528] LNet: 65032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.107411] LNetError: 65032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.117507] LNet: Removed LNI 192.168.201.44@tcp [ 1326.129697] LNet: Removed LNI 192.168.201.44@tcp1 [ 1326.607348] Key type .llcrypt unregistered [ 1326.611314] Key type ._llcrypt unregistered [ 1345.098520] Key type ._llcrypt registered [ 1345.104685] Key type .llcrypt registered [ 1346.545257] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:27:59 (1787495279) [ 1348.436989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1351.078124] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:28:03 (1787495283) [ 1352.583300] Key type .llcrypt unregistered [ 1352.585289] Key type ._llcrypt unregistered [ 1361.265652] Key type ._llcrypt registered [ 1361.269563] Key type .llcrypt registered [ 1361.562401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1361.597675] alg: No test for adler32 (adler32-zlib) [ 1362.554206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1362.579412] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1371.726973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1372.339628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e network_timeout -r 1 [ 1372.380589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e network_timeout -r 1 [ 1372.416841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1372.466138] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1372.819509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1372.838562] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1372.851728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1372.887598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1372.931109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1372.992762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.039795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1373.081480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1373.136826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1373.162084] LNet: 66581:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1373.203786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1373.308195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1373.395207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.515390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.612687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.681193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1374.206121] LNet: 66624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.222175] LNetError: 66624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.244654] LNet: Removed LNI 192.168.201.44@tcp [ 1375.026753] Key type .llcrypt unregistered [ 1375.028505] Key type ._llcrypt unregistered [ 1395.561176] Key type ._llcrypt registered [ 1395.569978] Key type .llcrypt registered [ 1395.794237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1395.809929] alg: No test for adler32 (adler32-zlib) [ 1396.771369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1396.799892] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1404.978445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1416.181555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.218048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1416.235435] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1417.348046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e network_timeout -r 1 [ 1417.379800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e network_timeout -r 1 [ 1417.412298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e network_timeout -r 1 [ 1417.464748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e network_timeout -r 1 [ 1417.509681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e network_timeout -r 1 [ 1417.592236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e network_timeout -r 1 [ 1417.658772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e network_timeout -r 1 [ 1417.710266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e network_timeout -r 1 [ 1417.762414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1417.798211] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1418.127195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1418.156893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1418.183182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1418.207798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1418.240782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1418.273130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1418.302324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1418.332707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1418.335787] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1418.337708] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1418.361751] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.144@tcp [ 1418.371355] LNet: 67509:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1418.392796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1418.444552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1418.472238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1418.501268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1418.540781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1418.574986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1423.557978] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1423.563328] LNet: Skipped 1 previous similar message [ 1423.923855] LNet: 67587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.935942] LNetError: 67587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.943948] LNet: Removed LNI 192.168.201.44@tcp [ 1423.954492] LNet: Removed LNI 192.168.201.44@tcp1 [ 1424.341410] Key type .llcrypt unregistered [ 1424.343580] Key type ._llcrypt unregistered [ 1441.473248] Key type ._llcrypt registered [ 1441.474945] Key type .llcrypt registered [ 1442.826757] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:29:35 (1787495375) [ 1443.243132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.255888] alg: No test for adler32 (adler32-zlib) [ 1444.181119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.295420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.346118] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1445.251889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.287692] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1446.332389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 1446.432287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1446.479886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1446.554906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1446.624318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1446.666837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1446.697305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.44@tcp [ 1446.730820] LNetError: 68687:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.44@tcp to recovery [ 1447.327345] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1447.328051] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1449.378728] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1450.246940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1453.471657] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1453.884786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1453.919809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.992240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.013583] LNet: 68794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.018655] LNetError: 68794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.071159] LNet: Removed LNI 192.168.201.44@tcp [ 1455.073050] LNetError: 68794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.093272] LNet: Removed LNI 192.168.201.44@tcp1 [ 1455.126407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.196342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.218662] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1456.110318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.130514] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1456.991269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 1457.024974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1457.090696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1457.153609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1457.211602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1457.243902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1457.291875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.329373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.44@tcp [ 1457.346628] LNetError: 69066:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.44@tcp to recovery [ 1458.208089] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1460.942336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.358479] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1464.373118] LNet: Skipped 1 previous similar message [ 1469.121920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1469.208200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1469.293613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.212645] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:30:09 (1787495409) [ 1476.316409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.351544] LNet: 69780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.374717] LNetError: 69780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.385397] LNet: Removed LNI 192.168.201.44@tcp [ 1476.430392] LNet: Removed LNI 192.168.201.44@tcp1 [ 1476.465076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.514296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.537625] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1477.267295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1477.288626] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1478.266418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 1478.324976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1478.347507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1478.385672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1478.416606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1478.443510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1478.468646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.44@tcp --health 0 [ 1478.556849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1478.567206] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.44@tcp [ 1479.583355] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1479.591990] LNet: Skipped 1 previous similar message [ 1494.208343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1494.358448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1494.384878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1494.408486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.44@tcp --health 500 [ 1494.492911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.305785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.383328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.417261] LNet: 70276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.445152] LNetError: 70276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.451596] LNetError: 70276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.459049] LNet: Removed LNI 192.168.201.44@tcp [ 1501.468597] LNet: Removed LNI 192.168.201.44@tcp1 [ 1501.486683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.564449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.601614] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1502.386740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.408474] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1503.482496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 1503.534749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1503.662977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1503.723656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1503.799892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1503.901046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1503.964077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1504.042133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.44@tcp --health 0 [ 1505.631858] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1505.651424] LNet: Skipped 3 previous similar messages [ 1506.597946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.289402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1517.350374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1517.389925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.619370] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:30:58 (1787495458) [ 1526.429748] LNet: 71333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.463278] LNetError: 71333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.475426] LNetError: 71333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1526.492527] LNet: Removed LNI 192.168.201.44@tcp [ 1526.507342] LNet: Removed LNI 192.168.201.44@tcp1 [ 1527.503746] Key type .llcrypt unregistered [ 1527.505856] Key type ._llcrypt unregistered [ 1535.530468] Key type ._llcrypt registered [ 1535.533123] Key type .llcrypt registered [ 1535.759872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1535.781199] alg: No test for adler32 (adler32-zlib) [ 1536.829936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1536.889433] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1547.069867] Lustre: DEBUG MARKER: Initial discovery [ 1547.344901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.144@tcp [ 1550.161194] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1550.253199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1550.346078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.144@tcp [ 1550.389538] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1550.478053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1550.553408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.144@tcp -r 1 -m GET -l 3 [ 1550.589775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.144@tcp -r 1 -m GET -e local_error [ 1550.620854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.144@tcp -r 1 -m PUT -l 6 [ 1551.264417] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1552.601832] Lustre: DEBUG MARKER: Force oleg144-client.virtnet to discover 192.168.201.144@tcp (in background) [ 1552.714893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.144@tcp [ 1554.520995] Lustre: DEBUG MARKER: Force oleg144-server to discover 192.168.201.44@tcp [ 1555.820267] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1557.142311] Lustre: DEBUG MARKER: Wait for 72095 [ 1559.918594] Lustre: DEBUG MARKER: Finished wait on 72095 [ 1559.974085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1560.055133] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1560.082334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1560.600414] LNet: 72555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.624631] LNetError: 72555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.715532] LNet: Removed LNI 192.168.201.44@tcp [ 1562.398693] Key type .llcrypt unregistered [ 1562.401833] Key type ._llcrypt unregistered [ 1580.064922] Key type ._llcrypt registered [ 1580.067512] Key type .llcrypt registered [ 1581.914422] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:31:54 (1787495514) [ 1582.481403] Key type .llcrypt unregistered [ 1582.483293] Key type ._llcrypt unregistered [ 1590.712340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1590.966681] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1590.970847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1590.975497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1592.787984] Key type ._llcrypt registered [ 1592.790431] Key type .llcrypt registered [ 1593.023812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.044624] alg: No test for adler32 (adler32-zlib) [ 1594.073914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.134953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.155945] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1595.017170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1595.037520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1604.559615] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:32:17 (1787495537) [ 1604.689772] LNet: 74622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.705883] LNetError: 74622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1604.728085] LNet: Removed LNI 10.1.2.3@tcp [ 1604.752725] LNet: Removed LNI 192.168.201.44@tcp [ 1605.493434] Key type .llcrypt unregistered [ 1605.497875] Key type ._llcrypt unregistered [ 1612.931541] Key type ._llcrypt registered [ 1612.941570] Key type .llcrypt registered [ 1613.252591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.285669] alg: No test for adler32 (adler32-zlib) [ 1614.221690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.323156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.361357] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1615.282341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1615.303526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1616.555540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.44@tcp [ 1617.631354] LNetError: Unexpected error -22 connecting to 192.168.201.44@tcp at host 192.168.201.44:988 [ 1623.176648] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:32:36 (1787495556) [ 1623.306622] LNet: 75771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.313756] LNetError: 75771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1623.323912] LNet: Removed LNI 10.1.2.3@tcp [ 1623.346552] LNet: Removed LNI 192.168.201.44@tcp [ 1623.979769] Key type .llcrypt unregistered [ 1623.983149] Key type ._llcrypt unregistered [ 1630.992745] Key type ._llcrypt registered [ 1630.994817] Key type .llcrypt registered [ 1631.178806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.195993] alg: No test for adler32 (adler32-zlib) [ 1632.054731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.109641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1632.133504] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1632.833496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1632.857264] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 1633.595673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.44@tcp1 --nid 192.168.201.44@tcp2 [ 1638.213910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp1 192.168.201.44@tcp2 [ 1638.230659] LNetError: 76618:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.44@tcp1 to 192.168.201.44@tcp2 - Route table may be misconfigured [ 1638.264689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp2 192.168.201.44@tcp1 [ 1645.443640] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:32:58 (1787495578) [ 1645.668061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.719734] LNet: 77191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.736981] LNetError: 77191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.751899] LNet: Removed LNI 192.168.201.44@tcp1 [ 1646.753992] LNetError: 77191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.776620] LNet: Removed LNI 192.168.201.44@tcp2 [ 1646.833462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.963942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.998772] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1647.898516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.928745] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1648.856741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 1648.924841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.44@tcp [ 1648.994412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp1 192.168.201.44@tcp1 [ 1649.072798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.44@tcp -d 192.168.201.44@tcp -e network_timeout [ 1649.124893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -e network_timeout [ 1649.173395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -e network_timeout [ 1649.223216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -e network_timeout [ 1649.296371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1649.338984] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1649.360897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1649.469528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.600147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1650.015288] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1657.005336] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:33:10 (1787495590) [ 1657.485441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.531593] LNet: 78053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.559758] LNetError: 78053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.591172] LNet: Removed LNI 192.168.201.44@tcp [ 1659.636862] LNet: Removed LNI 192.168.201.44@tcp1 [ 1659.724356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.827667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1660.261401] LNet: 78095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.753458] Key type .llcrypt unregistered [ 1660.756789] Key type ._llcrypt unregistered [ 1674.870366] Key type ._llcrypt registered [ 1674.873323] Key type .llcrypt registered [ 1676.749384] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:33:29 (1787495609) [ 1677.252453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.271686] alg: No test for adler32 (adler32-zlib) [ 1678.337772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.983365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1682.014836] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:33:34 (1787495614) [ 1682.318332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.357435] LNet: 79194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.399973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.463332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.500683] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1683.277939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1683.309290] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1684.412254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1684.453775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp1 [ 1684.498060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp1 [ 1690.848782] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:33:43 (1787495623) [ 1694.109616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg144-server [ 1695.957363] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:33:49 (1787495629) [ 1697.800218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg144-server [ 1700.549920] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:33:52 (1787495632) [ 1702.767847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg144-server [ 1705.386853] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:33:57 (1787495637) [ 1707.774415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg144-server [ 1710.407723] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:34:02 (1787495642) [ 1712.478349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg144-server [ 1714.409389] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:34:07 (1787495647) [ 1716.822893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg144-server [ 1718.833704] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:34:11 (1787495651) [ 1721.241415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg144-server [ 1723.703162] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:34:16 (1787495656) [ 1726.004874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg144-server [ 1728.713611] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:34:20 (1787495660) [ 1731.413127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg144-server [ 1733.791195] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:34:26 (1787495666) [ 1733.922858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.961716] LNet: 83068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.981755] LNetError: 83068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.997478] LNet: Removed LNI 192.168.201.44@tcp [ 1735.013523] LNet: Removed LNI 192.168.201.44@tcp1 [ 1735.094523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.227670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.310141] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1736.500046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1736.589446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1736.843790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.890169] LNet: 83221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.901168] LNetError: 83221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.910273] LNetError: 83221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1736.941226] LNet: Removed LNI 192.168.201.44@tcp [ 1736.979127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.045450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1737.084454] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1738.077406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1738.258760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.279365] LNetError: 83366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.291570] LNet: Removed LNI 192.168.201.44@tcp [ 1738.306754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.363370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.379323] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1738.959403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1739.049982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1739.206771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.225994] LNet: 83514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.231316] LNet: 83514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1740.261853] LNet: Removed LNI 192.168.201.44@tcp [ 1740.319375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.429235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1740.486630] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1741.300642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1741.519177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.548804] LNetError: 83659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.560194] LNetError: 83659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.637043] LNet: Removed LNI 192.168.201.44@tcp [ 1742.690870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.745600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.766301] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1743.644952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1743.802666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1744.194891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.258751] LNet: 83806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.264544] LNet: 83806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1745.322088] LNet: Removed LNI 192.168.201.44@tcp [ 1745.409782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.551581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1745.634753] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1746.903594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1747.100276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.115461] LNetError: 83952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.120160] LNetError: 83952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1747.138251] LNet: Removed LNI 192.168.201.44@tcp [ 1747.170474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.226344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.248624] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1748.113791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1748.173307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1748.440848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.557835] LNet: Removed LNI 192.168.201.44@tcp [ 1749.592534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.699720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1749.751710] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1750.720750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1751.058074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.167538] LNet: Removed LNI 192.168.201.44@tcp [ 1752.226984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.379070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.438271] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1753.607921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1753.746300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1754.053440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.103577] LNet: 84392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.112996] LNet: 84392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1755.183226] LNet: Removed LNI 192.168.201.44@tcp [ 1755.245306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.373890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1755.415483] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1756.617532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1757.010509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.049874] LNetError: 84537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.055313] LNetError: 84537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1758.118387] LNet: Removed LNI 192.168.201.44@tcp [ 1758.167983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.297327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.325767] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1759.145524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1759.268195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1759.722438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.807514] LNet: Removed LNI 192.168.201.44@tcp [ 1760.837139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.909668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1760.967631] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1762.260975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 1762.632423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.753185] LNet: Removed LNI 192.168.201.44@tcp [ 1763.835856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.924717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.967419] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1764.743311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1764.813542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.855624] LNet: Removed LNI 192.168.201.44@tcp [ 1764.874434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.913876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.950960] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1765.611264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1765.706807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.749763] LNet: Removed LNI 192.168.201.44@tcp [ 1765.789301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.865682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1765.966424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.035528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.106199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1766.186205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.215743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.261622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1766.330423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.359484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.405626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1766.423565] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1767.229718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.270362] LNet: Removed LNI 192.168.201.44@tcp [ 1767.292317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.375815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1774.412552] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:35:07 (1787495707) [ 1774.771575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.804030] LNet: 85854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.808526] LNet: 85854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1774.826274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.904485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1774.938353] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/180] [ 1776.305889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.342605] LNetError: 85992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.347032] LNetError: 85992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1776.370648] LNet: Removed LNI 192.168.201.44@tcp231 [ 1776.393614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.464139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1776.493888] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/180] [ 1776.510170] LNet: 86000:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1777.011162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1777.053974] LNet: Removed LNI 192.168.201.44@tcp231 [ 1777.080667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1777.127167] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/180] [ 1778.170352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.203519] LNet: Removed LNI 192.168.201.44@tcp231 [ 1778.247418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.325470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1778.342917] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/0] [ 1778.347081] LNet: 86152:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1778.835519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1778.863380] LNet: Removed LNI 192.168.201.44@tcp231 [ 1778.882215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1778.908128] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/0] [ 1779.943100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.982839] LNet: Removed LNI 192.168.201.44@tcp231 [ 1780.012379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.082023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1780.095781] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/60] [ 1780.100868] LNet: 86303:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.554868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1780.581764] LNet: Removed LNI 192.168.201.44@tcp231 [ 1780.598197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1780.627335] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/60] [ 1781.778647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.814475] LNet: Removed LNI 192.168.201.44@tcp231 [ 1781.861243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.948558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1781.986139] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/180] [ 1782.481363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1782.507602] LNet: Removed LNI 192.168.201.44@tcp231 [ 1782.523572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1782.546690] LNet: Added LNI 192.168.201.44@tcp231 [8/256/0/180] [ 1790.030986] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:35:23 (1787495723) [ 1790.166544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.205503] LNet: Removed LNI 192.168.201.44@tcp231 [ 1790.296716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.382063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1790.435273] LNet: Added LNI 192.168.201.44@tcp232 [8/256/0/180] [ 1791.008669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1791.045858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1791.072923] LNet: Removed LNI 192.168.201.44@tcp232 [ 1791.083694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1791.102903] LNet: Added LNI 192.168.201.44@tcp232 [8/256/0/180] [ 1791.841437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1791.872180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1791.916675] LNet: Removed LNI 192.168.201.44@tcp232 [ 1791.932474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1791.955410] LNet: Added LNI 192.168.201.44@tcp232 [8/256/0/180] [ 1792.545341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1792.573505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1792.606984] LNet: Removed LNI 192.168.201.44@tcp232 [ 1792.642247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1792.677752] LNet: Added LNI 192.168.201.44@tcp232 [8/256/0/180] [ 1793.411889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1793.477574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1793.522438] LNet: Removed LNI 192.168.201.44@tcp232 [ 1793.542329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1799.215175] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:35:32 (1787495732) [ 1800.010984] Key type .llcrypt unregistered [ 1800.012884] Key type ._llcrypt unregistered [ 1807.216522] Key type ._llcrypt registered [ 1807.220832] Key type .llcrypt registered [ 1807.556589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1807.590691] alg: No test for adler32 (adler32-zlib) [ 1808.538263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1808.559645] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1815.832724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1824.583679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.635154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1824.664158] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1826.302807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_interrupt -r 2 [ 1826.338566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_interrupt -r 2 [ 1826.391725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_interrupt -r 2 [ 1826.420268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_interrupt -r 2 [ 1826.456865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_interrupt -r 2 [ 1826.492088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_interrupt -r 2 [ 1826.535573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_interrupt -r 2 [ 1826.589690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_interrupt -r 2 [ 1826.645864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1826.679433] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1826.683831] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1826.991609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1827.028440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1827.081471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.126817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.169386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.199802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.239707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.277193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.304882] LNet: 88960:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.324414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1827.350577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1827.380297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.437430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1827.481476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1827.516561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.266394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_dropped -r 2 [ 1828.310742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_dropped -r 2 [ 1828.343724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_dropped -r 2 [ 1828.397460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_dropped -r 2 [ 1828.451569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_dropped -r 2 [ 1828.481361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_dropped -r 2 [ 1828.544226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_dropped -r 2 [ 1828.583159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_dropped -r 2 [ 1828.637606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1828.658200] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1828.967320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1829.001674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1829.046560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1829.097355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1829.147468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.180719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.218597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.260746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.284111] LNet: 89080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.336710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.387466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.473712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.533694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.580313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.626662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1830.268757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_aborted -r 2 [ 1830.328803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_aborted -r 2 [ 1830.393290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_aborted -r 2 [ 1830.436662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_aborted -r 2 [ 1830.480561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_aborted -r 2 [ 1830.553083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_aborted -r 2 [ 1830.617405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_aborted -r 2 [ 1830.652096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_aborted -r 2 [ 1830.712510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1830.772153] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1831.105135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1831.151978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1831.186470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.241727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.284664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.338993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.376154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.440647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.493396] LNet: 89201:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.511886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.554122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.616210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.661318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.689638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.731814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.329812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_no_route -r 2 [ 1832.385481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_no_route -r 2 [ 1832.454597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_no_route -r 2 [ 1832.509924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_no_route -r 2 [ 1832.575615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_no_route -r 2 [ 1832.646994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_no_route -r 2 [ 1832.713068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_no_route -r 2 [ 1832.787804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_no_route -r 2 [ 1832.842119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1832.880227] LNet: There was an unexpected network error while writing to 192.168.201.144: rc = -22 [ 1833.238781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.270500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1833.312981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.352461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.393956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.434463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.467772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.495091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.520377] LNet: 89321:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.534693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.556253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.594607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.631616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.675694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.716297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1834.317620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e local_timeout -r 2 [ 1834.357986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e local_timeout -r 2 [ 1834.403982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e local_timeout -r 2 [ 1834.446321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e local_timeout -r 2 [ 1834.505376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e local_timeout -r 2 [ 1834.547260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e local_timeout -r 2 [ 1834.584303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e local_timeout -r 2 [ 1834.632601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e local_timeout -r 2 [ 1834.701971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1834.992275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.034399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1835.075121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1835.104863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1835.138513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1835.171866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1835.211656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.248515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1835.281610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1835.306110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1835.334700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1835.364607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1835.401262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1835.437495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1835.927656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp -m GET -e remote_dropped -r 2 [ 1835.966747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.144@tcp1 -m GET -e remote_dropped -r 2 [ 1836.012302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp -m GET -e remote_dropped -r 2 [ 1836.050576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp -d 192.168.201.44@tcp1 -m GET -e remote_dropped -r 2 [ 1836.109455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp -m GET -e remote_dropped -r 2 [ 1836.170457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.44@tcp1 -d 192.168.201.144@tcp1 -m GET -e remote_dropped -r 2 [ 1836.211847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp -m GET -e remote_dropped -r 2 [ 1836.258207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.44@tcp1 -d 192.168.201.44@tcp1 -m GET -e remote_dropped -r 2 [ 1836.307405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.44@tcp 192.168.201.144@tcp [ 1836.342821] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.144@tcp [ 1836.567922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1836.594458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1836.623147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1836.648480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1836.667357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1836.695490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1836.728996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1836.762416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1836.796292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.835304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.865729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.895895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.936152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.978471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1842.352355] LNet: 89640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.356554] LNetError: 89640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.359190] LNet: Removed LNI 192.168.201.44@tcp [ 1843.362985] LNetError: 89640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.377540] LNet: Removed LNI 192.168.201.44@tcp1 [ 1843.955734] Key type .llcrypt unregistered [ 1843.957437] Key type ._llcrypt unregistered [ 1862.877906] Key type ._llcrypt registered [ 1862.880949] Key type .llcrypt registered [ 1864.277827] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:36:37 (1787495797) [ 1864.743160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.754427] alg: No test for adler32 (adler32-zlib) [ 1865.825062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.881459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.902405] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1866.698248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1866.721368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.44@tcp [ 1866.736405] LNetError: 90617:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.44@tcp to recovery [ 1866.814820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.44@tcp [ 1866.908412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1874.553764] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:36:47 (1787495807) [ 1874.692108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.719640] LNet: 91202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.735537] LNetError: 91202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.763201] LNet: Removed LNI 192.168.201.44@tcp [ 1874.790766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.862834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1874.911839] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1875.570858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1875.587964] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 1876.741748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.44@tcp [ 1876.766376] LNetError: 91474:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.44@tcp to recovery [ 1876.895397] LNet: 1 local NIs in recovery (showing 1): 192.168.201.44@tcp [ 1876.899465] LNet: There was an unexpected network error while writing to 192.168.201.44: rc = -22 [ 1878.061300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.44@tcp [ 1885.403309] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:36:58 (1787495818) [ 1887.884897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg144-server [ 1890.017168] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:37:02 (1787495822) [ 1892.497326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg144-server [ 1895.297617] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:37:07 (1787495827) [ 1897.530875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg144-server [ 1899.933134] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:37:12 (1787495832) [ 1900.014530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.034317] LNet: 93089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.040806] LNetError: 93089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.087419] LNet: Removed LNI 192.168.201.44@tcp [ 1901.092473] LNetError: 93089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.109536] LNet: Removed LNI 192.168.201.44@tcp1 [ 1901.145858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.275622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.300344] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1901.987460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1902.053564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1902.100470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1902.209822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1902.249789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1902.316949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1908.934358] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:37:21 (1787495841) [ 1909.089290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.138448] LNet: 93823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.144774] LNetError: 93823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.158712] LNet: Removed LNI 192.168.201.44@tcp [ 1909.188737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.274416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.301567] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1915.953569] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:37:29 (1787495849) [ 1916.200496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.217729] LNet: 94530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.223640] LNetError: 94530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.234688] LNet: Removed LNI 192.168.201.44@tcp [ 1916.246972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.310697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.358301] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1923.618898] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:37:36 (1787495856) [ 1925.107322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1927.176589] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:37:40 (1787495860) [ 1927.663876] LNet: 95565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.669073] LNetError: 95565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.679274] LNet: Removed LNI 192.168.201.44@tcp [ 1928.255913] Key type .llcrypt unregistered [ 1928.260509] Key type ._llcrypt unregistered [ 1934.544595] Key type ._llcrypt registered [ 1934.547438] Key type .llcrypt registered [ 1934.746980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1934.760610] alg: No test for adler32 (adler32-zlib) [ 1935.659183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1935.687318] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1943.514566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 1948.225081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.144@tcp [ 1948.587620] LNet: 95990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.595222] LNetError: 95990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.607189] LNet: Removed LNI 192.168.201.44@tcp [ 1949.105552] Key type .llcrypt unregistered [ 1949.107191] Key type ._llcrypt unregistered [ 1959.979771] Key type ._llcrypt registered [ 1959.982496] Key type .llcrypt registered [ 1961.589647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1963.206724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1964.714902] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:38:17 (1787495897) [ 1966.126474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1967.869764] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:38:20 (1787495900) [ 1969.377489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg144-server [ 1971.361567] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:38:24 (1787495904) [ 1973.983574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg144-server [ 1975.645564] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:38:28 (1787495908) [ 1976.982950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.127350] alg: No test for adler32 (adler32-zlib) [ 1978.341076] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1978.675892] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 1980.359553] Key type lgssc registered [ 1981.653604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.788955] Key type lgssc unregistered [ 2003.081050] LNet: 99166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.086963] LNetError: 99166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.108723] LNet: Removed LNI 192.168.201.44@tcp [ 2003.820763] Key type .llcrypt unregistered [ 2003.824459] Key type ._llcrypt unregistered [ 2019.687239] Key type ._llcrypt registered [ 2019.691649] Key type .llcrypt registered [ 2020.960297] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:39:13 (1787495953) [ 2021.271874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.287334] alg: No test for adler32 (adler32-zlib) [ 2022.185419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.236669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2022.317837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2022.373743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2022.419133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.290356] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:39:21 (1787495961) [ 2028.445259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.469553] LNet: 100581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.514059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.576100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2028.611072] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2029.368320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.384740] LNet: 100726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.389142] LNetError: 100726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.401550] LNet: Removed LNI 192.168.201.44@tcp [ 2029.424260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.468205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2029.490993] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2030.017747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.034164] LNetError: 100754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.057553] LNet: Removed LNI 192.168.201.44@tcp [ 2030.085294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.139200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2030.179859] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2036.296499] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:39:29 (1787495969) [ 2036.576473] LNet: 101343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.591228] LNet: 101343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2036.603364] LNetError: 101343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.620478] LNet: Removed LNI 192.168.201.44@tcp [ 2037.268756] Key type .llcrypt unregistered [ 2037.270314] Key type ._llcrypt unregistered [ 2044.059162] Key type ._llcrypt registered [ 2044.061423] Key type .llcrypt registered [ 2044.262591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2044.280227] alg: No test for adler32 (adler32-zlib) [ 2045.266413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.330423] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2045.333284] LNet: *** cfs_fail_loc=e003, val=100*** [ 2046.187921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2046.226481] LNet: *** cfs_fail_loc=e003, val=100*** [ 2046.229225] LNet: Skipped 2 previous similar messages [ 2046.234888] Ping reply received with mapping: [ 2046.238798] found metadata reply NI[192.168.201.44@tcp] [ 2046.241963] 2c c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2046.244556] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2046.246544] 5a 5a 5a 5a 5a 5a 5a 5a [ 2046.361811] LNet: 101745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.373639] LNetError: 101745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.385659] LNet: Removed LNI 192.168.201.44@tcp [ 2047.193520] Key type .llcrypt unregistered [ 2047.195461] Key type ._llcrypt unregistered [ 2058.039314] Key type ._llcrypt registered [ 2058.042510] Key type .llcrypt registered [ 2059.556418] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:39:52 (1787495992) [ 2060.053314] Key type .llcrypt unregistered [ 2060.057291] Key type ._llcrypt unregistered [ 2066.825330] Key type ._llcrypt registered [ 2066.827831] Key type .llcrypt registered [ 2066.982562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2066.992905] alg: No test for adler32 (adler32-zlib) [ 2068.225118] LNetError: 102802:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2068.658531] Key type .llcrypt unregistered [ 2068.661836] Key type ._llcrypt unregistered [ 2073.604814] Key type ._llcrypt registered [ 2073.607446] Key type .llcrypt registered [ 2074.926690] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:40:07 (1787496007) [ 2075.664547] Key type .llcrypt unregistered [ 2075.666545] Key type ._llcrypt unregistered [ 2081.206475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2081.211961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.881898] Key type ._llcrypt registered [ 2083.885981] Key type .llcrypt registered [ 2084.106498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.119992] alg: No test for adler32 (adler32-zlib) [ 2085.089731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.158346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2093.210833] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:40:26 (1787496026) [ 2093.285517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.311269] LNet: 104583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.316502] LNetError: 104583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2093.327553] LNet: Removed LNI 10.1.2.3@tcp [ 2093.341563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.926967] Key type .llcrypt unregistered [ 2093.928479] Key type ._llcrypt unregistered [ 2100.240561] Key type ._llcrypt registered [ 2100.242240] Key type .llcrypt registered [ 2100.463707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2100.478172] alg: No test for adler32 (adler32-zlib) [ 2101.484148] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2101.544510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2101.690360] LNet: 104885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.694415] LNetError: 104885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.758476] LNet: Removed LNI 192.168.201.44@tcp [ 2103.505535] Key type .llcrypt unregistered [ 2103.507092] Key type ._llcrypt unregistered [ 2116.077248] Key type ._llcrypt registered [ 2116.080966] Key type .llcrypt registered [ 2117.577399] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:40:50 (1787496050) [ 2118.271451] Key type .llcrypt unregistered [ 2118.273855] Key type ._llcrypt unregistered [ 2125.595872] Key type ._llcrypt registered [ 2125.597914] Key type .llcrypt registered [ 2125.822173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2125.832173] alg: No test for adler32 (adler32-zlib) [ 2134.549790] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:41:07 (1787496067) [ 2134.633092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.659976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.707549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.722550] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2135.245490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.392362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2135.439487] LNet: Removed LNI 192.168.201.44@tcp [ 2135.837779] LNet: 106642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.300752] Key type .llcrypt unregistered [ 2136.303361] Key type ._llcrypt unregistered [ 2149.751628] Key type ._llcrypt registered [ 2149.758821] Key type .llcrypt registered [ 2151.328430] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:41:23 (1787496083) [ 2151.887932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2151.902707] alg: No test for adler32 (adler32-zlib) [ 2152.982632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.053311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.085136] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2153.629421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2153.675398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.44@tcp [ 2153.689838] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2153.693346] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2153.696422] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2153, 0, 0 [ 2153.700946] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2153.703419] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2153.706575] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2153.709583] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2153, 0, 0 [ 2153.713944] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2153.716179] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2153.719334] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2153.722578] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2153, 0, 2203 [ 2153.726152] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2153.727794] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2153.730210] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2153.732022] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2153, 0, 2203 [ 2153.735961] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2160.337761] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:41:33 (1787496093) [ 2160.841292] LNet: 108183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.850257] LNetError: 108183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.873518] LNet: Removed LNI 192.168.201.44@tcp [ 2161.397399] Key type .llcrypt unregistered [ 2161.400330] Key type ._llcrypt unregistered [ 2167.762376] Key type ._llcrypt registered [ 2167.764640] Key type .llcrypt registered [ 2168.062803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2168.082125] alg: No test for adler32 (adler32-zlib) [ 2168.964444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2168.979227] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2176.372688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 2185.260388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.297234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2185.319081] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 2186.101394] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2186.242279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2186.251488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2186.258086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2188.565584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2188.584295] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2198.666990] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:42:11 (1787496131) [ 2199.130853] LNet: 109696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.145745] LNetError: 109696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.155645] LNet: Removed LNI 192.168.201.44@tcp [ 2200.225189] LNet: Removed LNI 192.168.201.44@tcp1 [ 2200.227509] LNetError: 109696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2200.236490] LNetError: 109696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2200.256026] LNet: Removed LNI 10.1.2.3@tcp99 [ 2201.101683] Key type .llcrypt unregistered [ 2201.105680] Key type ._llcrypt unregistered [ 2207.736528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2207.743936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2210.517700] Key type ._llcrypt registered [ 2210.520152] Key type .llcrypt registered [ 2210.696286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2210.714253] alg: No test for adler32 (adler32-zlib) [ 2211.601293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.644583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2211.664653] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2212.287979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2212.321765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2214.517320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2214.571299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2214.633878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2214.662808] LNetError: 110446:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2214.707064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2214.788693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2215.348420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2215.402498] LNet: 110466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.415062] LNetError: 110466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2215.426071] LNet: Removed LNI 10.1.2.3@tcp [ 2216.486694] LNet: Removed LNI 192.168.201.44@tcp [ 2217.253490] Key type .llcrypt unregistered [ 2217.257763] Key type ._llcrypt unregistered [ 2229.610841] Key type ._llcrypt registered [ 2229.618296] Key type .llcrypt registered [ 2230.724598] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:42:43 (1787496163) [ 2231.322604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.342510] alg: No test for adler32 (adler32-zlib) [ 2232.316718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.360241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.374810] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2233.108990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2233.194753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg144-client@tcp [ 2239.568203] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:42:52 (1787496172) [ 2239.676633] LNet: 111996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.687978] LNetError: 111996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.700648] LNet: Removed LNI 192.168.201.44@tcp [ 2240.267475] Key type .llcrypt unregistered [ 2240.273798] Key type ._llcrypt unregistered [ 2246.525219] Key type ._llcrypt registered [ 2246.526907] Key type .llcrypt registered [ 2246.644783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.654592] alg: No test for adler32 (adler32-zlib) [ 2247.625469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2247.660560] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2247.713875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2253.972613] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:43:06 (1787496186) [ 2255.794560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2257.561522] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:43:10 (1787496190) [ 2258.778936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2260.852941] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:43:13 (1787496193) [ 2261.297558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.356652] LNet: 113430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.365169] LNetError: 113430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.387788] LNet: Removed LNI 192.168.201.44@tcp [ 2261.415227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.505987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2261.531858] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2262.531271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2262.600118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2270.140860] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:43:22 (1787496202) [ 2270.473531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.506323] LNet: 114139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.516088] LNetError: 114139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.533699] LNet: Removed LNI 192.168.201.44@tcp [ 2270.580584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.669507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2270.719361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2271.629343] Key type .llcrypt unregistered [ 2271.632323] Key type ._llcrypt unregistered [ 2284.070137] Key type ._llcrypt registered [ 2284.075113] Key type .llcrypt registered [ 2285.741868] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:43:38 (1787496218) [ 2286.329156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.353807] alg: No test for adler32 (adler32-zlib) [ 2287.280955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.329507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2287.351674] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2288.204757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2288.304577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2294.167691] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:43:47 (1787496227) [ 2294.463945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.492224] LNet: 115699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.497596] LNetError: 115699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.518095] LNet: Removed LNI 192.168.201.44@tcp [ 2294.552584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.621642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2294.665362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2301.202634] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:43:53 (1787496233) [ 2301.408009] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.460468] LNet: 116278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.520030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.641479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2302.009455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2302.039392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2302.103967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2302.492558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2302.520444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2302.574961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.592309] LNet: 116313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.619594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.731266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2303.293534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2303.351235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2303.444121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2303.992270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2304.045681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2312.507685] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:44:05 (1787496245) [ 2312.614512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.676056] LNet: 116907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.724996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.902657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.940978] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2313.984220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2314.017336] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2315.175118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2315.247069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2315.618977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.649562] LNetError: 117193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.707330] LNet: Removed LNI 192.168.201.44@tcp [ 2316.720719] LNetError: 117193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.764141] LNet: Removed LNI 192.168.201.44@tcp2 [ 2316.814172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.922076] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2316.932611] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2316.941745] LNet: 117199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2317.718891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2318.013590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.033459] LNet: 117237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.043264] LNet: 117237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2319.071269] LNet: Removed LNI 192.168.201.44@tcp [ 2319.073263] LNetError: 117237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.098602] LNetError: 117237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2319.135844] LNet: Removed LNI 192.168.201.44@tcp2 [ 2319.192990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.289855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.326097] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2320.133771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2320.157819] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2321.324635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2321.406749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2321.504580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2321.860948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.912620] LNet: Removed LNI 192.168.201.44@tcp [ 2323.954432] LNet: Removed LNI 192.168.201.44@tcp2 [ 2324.006246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.104742] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2324.111487] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2324.117877] LNet: 117527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.809096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2325.007860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.028766] LNetError: 117565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.034354] LNetError: 117565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2325.041639] LNet: Removed LNI 192.168.201.44@tcp [ 2326.053795] LNet: Removed LNI 192.168.201.44@tcp2 [ 2326.127478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.204584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.241031] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2326.984056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2327.009762] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2327.660270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2327.783786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2328.016859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.033963] LNet: 117844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.038992] LNet: 117844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2329.061205] LNet: Removed LNI 192.168.201.44@tcp [ 2329.073920] LNet: Removed LNI 192.168.201.44@tcp2 [ 2329.127633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.244089] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2329.257432] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2329.264653] LNet: 117850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2329.568380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2329.791879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.848897] LNet: Removed LNI 192.168.201.44@tcp [ 2330.866238] LNet: Removed LNI 192.168.201.44@tcp2 [ 2330.894564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.001385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.047805] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2332.054340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2332.090502] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2333.196808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2333.226964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2333.274996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2333.540264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.568008] LNetError: 118172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.582088] LNetError: 118172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2333.587417] LNet: Removed LNI 192.168.201.44@tcp [ 2334.634093] LNet: Removed LNI 192.168.201.44@tcp2 [ 2334.654689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.754470] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2334.767664] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2334.771257] LNet: 118178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.338363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2335.635661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.668901] LNet: Removed LNI 192.168.201.44@tcp [ 2336.743637] LNet: Removed LNI 192.168.201.44@tcp2 [ 2336.777548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.869451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.905586] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2337.915902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2337.965330] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2338.889391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2338.925993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2339.028912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2339.662273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.703468] LNet: Removed LNI 192.168.201.44@tcp [ 2340.718790] LNet: Removed LNI 192.168.201.44@tcp2 [ 2340.773388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.914929] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2340.927407] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2340.945643] LNet: 118518:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.829559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2342.186618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.263907] LNet: Removed LNI 192.168.201.44@tcp [ 2343.281639] LNet: Removed LNI 192.168.201.44@tcp2 [ 2343.331723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.441508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.473326] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2344.310730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2344.332080] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2345.249281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2345.298568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2345.349035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2345.403897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2345.786865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.809482] LNet: 118872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.814645] LNet: 118872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2346.847165] LNet: Removed LNI 192.168.201.44@tcp [ 2347.878236] LNet: Removed LNI 192.168.201.44@tcp2 [ 2347.915810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.990191] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2347.995429] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2348.524136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2356.540553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2358.005345] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:44:51 (1787496291) [ 2358.140844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.170356] LNetError: 119632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.183091] LNetError: 119632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2358.194416] LNet: Removed LNI 192.168.201.44@tcp [ 2359.275188] LNet: Removed LNI 192.168.201.44@tcp2 [ 2359.310555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.634173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.653454] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2360.454879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2360.476284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2361.290981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.332789] LNet: Removed LNI 10.1.2.3@tcp [ 2361.346284] LNet: Removed LNI 192.168.201.44@tcp [ 2361.395775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.538948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2361.588040] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2361.595460] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2362.084174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2362.101264] LNet: Removed LNI 192.168.201.44@tcp [ 2362.116921] LNet: Removed LNI 10.1.2.3@tcp [ 2369.376191] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:45:02 (1787496302) [ 2369.542145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.604270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.722911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.790619] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2370.769538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2370.806615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2370.838536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2370.870341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2370.908202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2370.945128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2370.982752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2371.013471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2371.043968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2371.079845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2371.134414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.44@tcp [ 2371.315255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2378.611111] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:45:11 (1787496311) [ 2379.022686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.068572] LNet: 121253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.080451] LNet: 121253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2380.135832] LNet: Removed LNI 192.168.201.44@tcp [ 2380.169472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.358518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2380.518106] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.536048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2380.548855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2382.920927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.959147] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2383.677826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2383.695647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2385.434447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2385.470606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2385.519343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.44@tcp -m PUT -e local_timeout -r 1 [ 2385.580473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2385.629347] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2385.636744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2385.850872] LNetError: 121311:0:(lib-move.c:4801:lnet_parse()) 192.168.201.44@tcp, src 192.168.201.44@tcp: Dropping PUT (error -108 looking up sender) [ 2386.918076] LNet: Removed LNI 10.1.2.3@tcp [ 2387.941916] LNet: Removed LNI 192.168.201.44@tcp [ 2388.392444] Key type .llcrypt unregistered [ 2388.395687] Key type ._llcrypt unregistered [ 2400.912913] Key type ._llcrypt registered [ 2400.926323] Key type .llcrypt registered [ 2402.253347] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:45:35 (1787496335) [ 2402.917624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.932526] alg: No test for adler32 (adler32-zlib) [ 2403.818418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.160521] LNet: 122629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2404.725639] Key type .llcrypt unregistered [ 2404.729472] Key type ._llcrypt unregistered [ 2411.998228] Key type ._llcrypt registered [ 2412.005419] Key type .llcrypt registered [ 2412.165437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2412.178323] alg: No test for adler32 (adler32-zlib) [ 2413.104912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2413.128597] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2421.283133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 2421.477847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2422.105945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.144@tcp [ 2422.367088] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.144@tcp [ 2422.395652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.461365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.544356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.585431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.615180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.683800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.717205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.750919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.770618] LNet: 123083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.789709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.821474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.856938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.909931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.949490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.011511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.739364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2424.019129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.073163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.117325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.159083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.211400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.263816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.312428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.346327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.368844] LNet: 123201:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2424.403685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.440967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.495685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.540968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.573771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.624745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.250591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.144@tcp [ 2425.521153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.565607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.600526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.630978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.656711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.683381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.713658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.749767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.770585] LNet: 123318:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2425.794581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.841045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.861631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.893880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.953842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.988784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2426.630326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2426.882909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2426.933859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2426.971074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2427.022992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2427.062386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2427.103707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2427.159270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2427.198273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2427.230344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2427.277940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2427.307632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2427.339439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2427.370781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2427.405628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.684202] LNet: 123485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.691265] LNetError: 123485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.716235] LNet: Removed LNI 192.168.201.44@tcp [ 2428.023704] Key type .llcrypt unregistered [ 2428.025067] Key type ._llcrypt unregistered [ 2445.611368] Key type ._llcrypt registered [ 2445.613690] Key type .llcrypt registered [ 2446.748775] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:46:19 (1787496379) [ 2447.294976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.308954] alg: No test for adler32 (adler32-zlib) [ 2448.291771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2448.831739] LNet: 124373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.569776] Key type .llcrypt unregistered [ 2449.572732] Key type ._llcrypt unregistered [ 2456.176152] Key type ._llcrypt registered [ 2456.180154] Key type .llcrypt registered [ 2456.383509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2456.401672] alg: No test for adler32 (adler32-zlib) [ 2457.508725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2457.584586] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2466.300352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.144@tcp [ 2476.124715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.190881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2476.228844] LNet: Added LNI 192.168.201.44@tcp1 [8/256/0/180] [ 2476.921339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2477.463231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.144@tcp [ 2477.660083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2477.680878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2477.713740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2477.764892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2477.807460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.860339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.921428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2477.961764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2477.992215] LNet: 125031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.024918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2478.075736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2478.107060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2478.130916] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.144@tcp [ 2478.177751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2478.237792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2478.268827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.896764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2479.100819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.127978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.161762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.186685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.216279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.241776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.275386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.304497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.318721] LNet: 125148:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.334957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2479.362740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2479.388659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2479.419495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2479.453460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2479.493545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.166981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.144@tcp [ 2480.434508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2480.461795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2480.488478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2480.517612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2480.540483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2480.565798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2480.590502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2480.621629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2480.636883] LNet: 125266:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2480.657216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.683388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.708785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.735494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.771183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.798087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2481.313800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2481.566894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.593885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2481.629261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.658675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.682444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.707988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.739582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.762289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.787928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.812402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.840975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.869295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.895493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2481.931178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.942355] LNet: 125468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2486.949352] LNetError: 125468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2486.954966] LNet: Removed LNI 192.168.201.44@tcp [ 2486.971270] LNet: Removed LNI 192.168.201.44@tcp1 [ 2487.382683] Key type .llcrypt unregistered [ 2487.389305] Key type ._llcrypt unregistered [ 2503.894600] Key type ._llcrypt registered [ 2503.899741] Key type .llcrypt registered [ 2505.008937] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:47:18 (1787496438) [ 2505.460742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.488712] alg: No test for adler32 (adler32-zlib) [ 2506.426807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.498606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2506.537505] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2507.513762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2507.535719] LNet: Added LNI 192.168.201.44@tcp2 [8/256/0/180] [ 2508.691755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.44@tcp [ 2508.745436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2509.791344] LNet: Removed LNI 192.168.201.44@tcp [ 2516.981281] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:47:29 (1787496449) [ 2518.655879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2520.500729] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:47:33 (1787496453) [ 2528.549039] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:47:41 (1787496461) [ 2537.114979] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:47:49 (1787496469) [ 2544.654681] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:47:57 (1787496477) [ 2553.818267] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:48:06 (1787496486) [ 2562.592578] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:48:15 (1787496495) [ 2570.696690] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:48:23 (1787496503) [ 2578.548274] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:48:31 (1787496511) [ 2588.471981] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:48:40 (1787496520) [ 2599.010112] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:48:51 (1787496531) [ 2606.644788] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:48:59 (1787496539) [ 2614.400980] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:49:06 (1787496546) [ 2621.816287] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:49:14 (1787496554) [ 2628.757204] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:49:21 (1787496561) [ 2635.874203] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:49:28 (1787496568) [ 2644.730330] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:49:37 (1787496577) [ 2655.171687] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:49:47 (1787496587) [ 2664.372567] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:49:57 (1787496597) [ 2664.877164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2664.919691] LNet: 137188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.942756] LNetError: 137188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.984454] LNet: Removed LNI 192.168.201.44@tcp2 [ 2665.045212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2665.328637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2665.360941] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2666.822745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2666.855395] LNet: 137331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.866974] LNetError: 137331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.895162] LNet: Removed LNI 192.168.201.44@tcp [ 2666.925607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2666.987806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2667.011671] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2673.833921] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:50:06 (1787496606) [ 2681.263667] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:50:14 (1787496614) [ 2689.543730] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:50:22 (1787496622) [ 2698.879848] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:50:31 (1787496631) [ 2706.999315] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:50:39 (1787496639) [ 2714.344229] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:50:47 (1787496647) [ 2721.726453] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:50:54 (1787496654) [ 2729.372222] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:51:02 (1787496662) [ 2737.386665] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:51:10 (1787496670) [ 2745.311746] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:51:18 (1787496678) [ 2752.827768] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:51:25 (1787496685) [ 2761.460794] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:51:33 (1787496693) [ 2771.120334] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2482 sec ========== 10:51:43 (1787496703) [ 2771.418355] LNet: 144795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.429358] LNetError: 144795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.460821] LNet: Removed LNI 192.168.201.44@tcp [ 2772.159457] Key type .llcrypt unregistered [ 2772.164891] Key type ._llcrypt unregistered [ 2786.137754] Key type ._llcrypt registered [ 2786.140602] Key type .llcrypt registered [ 2786.843070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2786.855625] alg: No test for adler32 (adler32-zlib) [ 2787.996702] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2788.233690] LNet: Added LNI 192.168.201.44@tcp [8/256/0/180] [ 2789.911894] Key type lgssc registered [ 2791.721690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.461090] Lustre: Mounted lustre-client - version 2.17.57_80_gd2e8623 [ 2869.966726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2879.628293] Lustre: Unmounted lustre-client [ 2900.587173] Key type lgssc unregistered [ 2900.949355] LNet: 147758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2900.959187] LNetError: 147758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2900.969390] LNet: Removed LNI 192.168.201.44@tcp [ 2901.900197] Key type .llcrypt unregistered [ 2901.901749] Key type ._llcrypt unregistered