[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 448358495 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002376] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007724] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011038] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022669] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026697] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.031434] *** VALIDATE proc *** [ 0.033125] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042172] debug: unmapping init [mem 0xffffffff9ae59000-0xffffffff9ae60fff] [ 0.044169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045662] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053321] rcu: Hierarchical SRCU implementation. [ 0.055378] smp: Bringing up secondary CPUs ... [ 0.056585] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.061634] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139029] node 0 deferred pages initialised in 73ms [ 0.142519] devtmpfs: initialized [ 0.143243] x86/mm: Memory block size: 128MB [ 0.145707] gcov: version magic: 0x41383552 [ 0.148300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155454] pinctrl core: initialized pinctrl subsystem [ 0.158213] [ 0.158901] ************************************************************* [ 0.160012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161012] ** ** [ 0.164012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166013] ** ** [ 0.169017] ** This means that this kernel is built to expose internal ** [ 0.171012] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.176012] ** ** [ 0.179014] ** If you see this message and you are not debugging the ** [ 0.181010] ** kernel, report this immediately to your vendor! ** [ 0.183016] ** ** [ 0.186014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188010] ************************************************************* [ 0.190815] NET: Registered protocol family 16 [ 0.192452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201095] cpuidle: using governor menu [ 0.202748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204384] PCI: Using configuration type 1 for base access [ 0.206164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213163] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216091] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219224] cryptd: max_cpu_qlen set to 1000 [ 0.222319] ACPI: Added _OSI(Module Device) [ 0.224067] ACPI: Added _OSI(Processor Device) [ 0.226010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232599] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237502] ACPI: Interpreter enabled [ 0.239070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241024] ACPI: Using IOAPIC for interrupt routing [ 0.242107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256547] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269301] acpiphp: Slot [2] registered [ 0.271146] acpiphp: Slot [5] registered [ 0.272110] acpiphp: Slot [6] registered [ 0.273081] acpiphp: Slot [3] registered [ 0.275156] acpiphp: Slot [4] registered [ 0.276124] acpiphp: Slot [7] registered [ 0.278130] acpiphp: Slot [8] registered [ 0.279197] acpiphp: Slot [9] registered [ 0.281115] acpiphp: Slot [10] registered [ 0.282091] acpiphp: Slot [11] registered [ 0.283000] acpiphp: Slot [12] registered [ 0.283000] acpiphp: Slot [13] registered [ 0.285117] acpiphp: Slot [14] registered [ 0.286169] acpiphp: Slot [15] registered [ 0.288131] acpiphp: Slot [16] registered [ 0.290132] acpiphp: Slot [17] registered [ 0.291173] acpiphp: Slot [18] registered [ 0.292117] acpiphp: Slot [19] registered [ 0.294108] acpiphp: Slot [20] registered [ 0.295063] acpiphp: Slot [21] registered [ 0.296138] acpiphp: Slot [22] registered [ 0.298091] acpiphp: Slot [23] registered [ 0.299113] acpiphp: Slot [24] registered [ 0.301113] acpiphp: Slot [25] registered [ 0.303173] acpiphp: Slot [26] registered [ 0.304166] acpiphp: Slot [27] registered [ 0.306118] acpiphp: Slot [28] registered [ 0.307097] acpiphp: Slot [29] registered [ 0.309101] acpiphp: Slot [30] registered [ 0.310134] acpiphp: Slot [31] registered [ 0.311058] PCI host bridge to bus 0000:00 [ 0.312028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328017] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331340] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.345060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357098] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359869] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366642] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.381693] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386617] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391568] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431706] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467104] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.469394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.472397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.473513] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.475288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.478187] iommu: Default domain type: Passthrough [ 0.480593] SCSI subsystem initialized [ 0.482138] ACPI: bus type USB registered [ 0.484156] usbcore: registered new interface driver usbfs [ 0.486124] usbcore: registered new interface driver hub [ 0.488101] usbcore: registered new device driver usb [ 0.490239] pps_core: LinuxPPS API ver. 1 registered [ 0.491008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.493087] PTP clock support registered [ 0.495137] EDAC MC: Ver: 3.0.0 [ 0.497119] PCI: Using ACPI for IRQ routing [ 0.498795] NetLabel: Initializing [ 0.501014] NetLabel: domain hash size = 128 [ 0.502014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.504089] NetLabel: unlabeled traffic allowed by default [ 0.506112] vgaarb: loaded [ 0.508401] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.511027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.517465] clocksource: Switched to clocksource kvm-clock [ 0.625229] VFS: Disk quotas dquot_6.6.0 [ 0.626609] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.628698] *** VALIDATE ramfs *** [ 0.629672] *** VALIDATE hugetlbfs *** [ 0.631186] pnp: PnP ACPI init [ 0.633375] pnp: PnP ACPI: found 6 devices [ 0.657741] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.662990] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.665700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.667807] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.670690] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.673599] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.676335] NET: Registered protocol family 2 [ 0.678992] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.684842] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.688587] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.694374] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.698859] TCP: Hash tables configured (established 65536 bind 65536) [ 0.702372] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.706466] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.709138] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.712406] NET: Registered protocol family 1 [ 0.714921] RPC: Registered named UNIX socket transport module. [ 0.717492] RPC: Registered udp transport module. [ 0.719424] RPC: Registered tcp transport module. [ 0.721301] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.724101] NET: Registered protocol family 44 [ 0.725888] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.728440] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.730697] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.733290] PCI: CLS 0 bytes, default 64 [ 0.735770] Unpacking initramfs... [ 2.204606] debug: unmapping init [mem 0xffff94d47cc64000-0xffff94d47ffcffff] [ 2.209207] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.212018] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.215206] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.826108] Initialise system trusted keyrings [ 2.831233] Key type blacklist registered [ 2.835480] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.848086] zbud: loaded [ 2.851805] *** VALIDATE nfs *** [ 2.853854] *** VALIDATE nfs4 *** [ 2.856078] pstore: using deflate compression [ 2.860630] Platform Keyring initialized [ 2.974370] NET: Registered protocol family 38 [ 2.976619] Key type asymmetric registered [ 2.978774] Asymmetric key parser 'x509' registered [ 2.980912] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.984486] io scheduler mq-deadline registered [ 2.985983] io scheduler kyber registered [ 2.987335] io scheduler bfq registered [ 2.988955] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991740] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.994271] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.996748] ACPI: Power Button [PWRF] [ 3.003973] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.011846] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.029035] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.057970] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.086864] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.091529] Non-volatile memory driver v1.3 [ 3.093804] Linux agpgart interface v0.103 [ 3.132292] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.137286] vda: detected capacity change from 0 to 76677120 [ 3.159043] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.162364] vdb: detected capacity change from 0 to 1073741824 [ 3.170511] libphy: Fixed MDIO Bus: probed [ 3.177787] usbcore: registered new interface driver usbserial_generic [ 3.180571] usbserial: USB Serial support registered for generic [ 3.183454] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.188122] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.190338] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.192929] mousedev: PS/2 mouse device common for all mice [ 3.196494] rtc_cmos 00:05: RTC can wake from S4 [ 3.200877] rtc_cmos 00:05: registered as rtc0 [ 3.201235] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202586] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.208050] intel_pstate: CPU model not supported [ 3.211190] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.213879] hid: raw HID events driver (C) Jiri Kosina [ 3.218151] usbcore: registered new interface driver usbhid [ 3.220715] usbhid: USB HID core driver [ 3.220926] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.222111] drop_monitor: Initializing network drop monitor service [ 3.227055] Initializing XFRM netlink socket [ 3.229568] NET: Registered protocol family 10 [ 3.233348] Segment Routing with IPv6 [ 3.235176] NET: Registered protocol family 17 [ 3.237918] mpls_gso: MPLS GSO support [ 3.248115] RAS: Correctable Errors collector initialized. [ 3.250342] AVX version of gcm_enc/dec engaged. [ 3.252326] AES CTR mode by8 optimization enabled [ 3.350443] sched_clock: Marking stable (3350415332, 0)->(4259146305, -908730973) [ 3.355501] registered taskstats version 1 [ 3.357823] Loading compiled-in X.509 certificates [ 3.359871] zswap: loaded using pool lzo/zbud [ 3.387496] Key type big_key registered [ 3.402837] Key type encrypted registered [ 3.404381] ima: No TPM chip found, activating TPM-bypass! [ 3.406129] ima: Allocated hash algorithm: sha1 [ 3.407607] ima: No architecture policies found [ 3.409497] evm: Initialising EVM extended attributes: [ 3.411420] evm: security.selinux [ 3.412412] evm: security.ima [ 3.413460] evm: security.capability [ 3.414546] evm: HMAC attrs: 0x1 [ 3.416657] rtc_cmos 00:05: setting system clock to 2026-09-05 15:25:59 UTC (1788621959) [ 3.423104] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.428516] debug: unmapping init [mem 0xffffffff9ab82000-0xffffffff9ae58fff] [ 3.441085] Write protecting the kernel read-only data: 28672k [ 3.444116] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.446685] debug: unmapping init [mem 0xffffffff99b14000-0xffffffff99bfffff] [ 3.477319] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.484797] systemd[1]: Detected virtualization kvm. [ 3.486721] systemd[1]: Detected architecture x86-64. [ 3.488905] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.514806] systemd[1]: No hostname configured. [ 3.516824] systemd[1]: Set hostname to . [ 3.519317] random: systemd: uninitialized urandom read (16 bytes read) [ 3.522124] systemd[1]: Initializing machine ID from random generator. [ 3.568840] random: ln: uninitialized urandom read (6 bytes read) [ 3.725892] random: systemd: uninitialized urandom read (16 bytes read) [ 3.730691] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.737036] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.740611] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.357826] device-mapper: uevent: version 1.0.3 [ 4.360495] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.059756] virtio_net virtio0 ens2: renamed from eth0 [ 5.153057] scsi host0: ata_piix [ 5.184785] scsi host1: ata_piix [ 5.186545] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.189597] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.025200] random: fast init done [ 8.649279] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.937884] random: crng init done [ 9.939591] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.262066] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.428214] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.688377] SELinux: Disabled at runtime. [ 11.755545] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.765659] systemd[1]: Detected virtualization kvm. [ 11.768187] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.337807] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.341970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.349838] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.356464] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.360401] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.370637] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.374735] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ 12.549794] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.925352] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.209233] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.287869] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.356433] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.373322] EDAC sbridge: Ver: 1.1.2 [ 14.529862] Key type dns_resolver registered [ 14.843984] NFS: Registering the id_resolver key type [ 14.847674] Key type id_resolver registered [ 14.849243] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-client login: [ 47.844629] libcfs: loading out-of-tree module taints kernel. [ 48.068377] Key type ._llcrypt registered [ 48.094832] Key type .llcrypt registered [ 48.948279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.975975] alg: No test for adler32 (adler32-zlib) [ 50.660760] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 51.691856] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 53.535219] Key type lgssc registered [ 55.604505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.430350] hrtimer: interrupt took 11341070 ns [ 199.306382] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da [ 205.063223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.955678] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.736412] Lustre: lustre-OST0000-osc-ffff94d4ca761800: disconnect after 22s idle [ 227.545302] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing yml_node [ 232.893597] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 235.840500] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 238.502756] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 240.483517] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 11:29:54 EDT 2026 [ 257.279411] Lustre: Unmounted lustre-client [ 277.821871] Key type lgssc unregistered [ 278.144217] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.152646] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.177117] LNet: Removed LNI 192.168.204.46@tcp [ 279.173289] Key type .llcrypt unregistered [ 279.175673] Key type ._llcrypt unregistered [ 309.862800] Key type ._llcrypt registered [ 309.864553] Key type .llcrypt registered [ 310.112622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 310.144211] alg: No test for adler32 (adler32-zlib) [ 311.098982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 311.188577] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 311.211601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 316.482534] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 316.779637] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.810595] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.843509] LNet: Removed LNI 192.168.204.46@tcp [ 317.642779] Key type .llcrypt unregistered [ 317.646538] Key type ._llcrypt unregistered [ 325.893341] Key type ._llcrypt registered [ 325.896623] Key type .llcrypt registered [ 328.206571] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:31:21 (1788622281) [ 328.691648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.706311] alg: No test for adler32 (adler32-zlib) [ 329.676227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.827533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.869461] LNet: 9708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.896469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.012035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.064463] LNet: 9715:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 339.315190] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:31:33 (1788622293) [ 339.439786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.477363] LNet: 10289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.513116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.621476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 346.437080] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:31:41 (1788622301) [ 346.540801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.570235] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.604220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.752715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 354.597416] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:31:49 (1788622309) [ 354.811792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.858247] LNet: 11468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.880741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.997332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 362.633688] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:31:57 (1788622317) [ 362.769919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.815778] LNet: 12048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.853871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.003317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 363.770399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.990552] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:32:06 (1788622326) [ 372.167818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.233572] LNet: 12637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.284744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.439447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 373.468506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.613094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 383.957447] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:32:18 (1788622338) [ 384.087218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.146450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.300989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 392.932941] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:32:26 (1788622346) [ 393.057778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.116318] LNet: 13819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.129179] LNet: 13819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 393.166090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.339963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.398812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.046086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 394.093663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.728483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.779761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.225364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.303029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.804982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.853749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.261779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.303888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.802210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.843515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.396167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.455412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.928081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.979201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.399436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 398.428163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.777392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 398.816169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.209950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.261915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.917504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 399.992329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.686292] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:32:42 (1788622362) [ 407.916798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.993816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.091164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 408.143632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 415.952634] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:32:50 (1788622370) [ 416.172319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.237529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.354929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.414899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 424.518931] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:32:59 (1788622379) [ 424.754857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.823471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.916490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 424.979779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 432.197851] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:33:06 (1788622386) [ 432.371522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.414547] LNet: 16262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.427195] LNet: 16262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 432.536581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.650358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 432.690975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 441.235540] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:33:14 (1788622394) [ 441.420337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.507274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.659693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 441.695989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 450.277214] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:33:24 (1788622404) [ 450.477879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.526826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.658351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.703823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.284613] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:33:34 (1788622414) [ 460.429590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.506825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.713390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.140087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.060067] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:33:43 (1788622423) [ 469.275816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.364479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.630094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.064757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.249579] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:33:53 (1788622433) [ 479.470093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.535739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.659759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.922883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 487.404930] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:34:01 (1788622441) [ 487.818585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.923394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.556451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.924880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.875953] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:34:11 (1788622451) [ 497.310222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.349441] LNet: 20510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.368352] LNet: 20510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 497.391394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.460775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.502695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.029300] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:34:19 (1788622459) [ 505.184093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.262907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.396495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.456536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.432787] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:34:27 (1788622467) [ 513.646718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.700787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.782867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 521.281203] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:34:35 (1788622475) [ 521.406382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.480343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.607919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 528.666681] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:34:43 (1788622483) [ 528.847875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.897385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.981755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 529.398257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 536.859586] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:34:51 (1788622491) [ 536.925996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.974409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.075458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 537.113390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 544.049534] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:34:58 (1788622498) [ 544.146667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.227336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.341394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 544.404948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 551.884152] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:35:06 (1788622506) [ 552.241818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.289498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.393865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 552.449996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 560.686221] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:35:14 (1788622514) [ 560.994156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.058333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.182715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 561.275445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.314210] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:35:23 (1788622523) [ 568.679927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.739533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.837539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 568.967896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 568.991570] LNetError: 25827:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 569.030913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 569.059355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 575.293482] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:35:29 (1788622529) [ 575.423132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.511201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.675862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B154XZihcz [ 575.767446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B154XZihcz [ 575.875105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.B154XZihcz [ 582.558837] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:35:37 (1788622537) [ 582.788308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.814406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.875026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 582.936403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.018188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 589.492915] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 11:35:44 (1788622544) [ 589.800367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.853654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.937707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.961919] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 591.341157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 592.502093] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 594.463196] Key type lgssc registered [ 595.564396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.176946] LNetError: 27589:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.46@tcp99 has no available nets [ 601.833115] Key type lgssc unregistered [ 602.173371] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.188608] LNet: Removed LNI 192.168.204.46@tcp [ 602.876096] Key type .llcrypt unregistered [ 602.878756] Key type ._llcrypt unregistered [ 609.469160] Key type ._llcrypt registered [ 609.470972] Key type .llcrypt registered [ 611.111346] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:36:05 (1788622565) [ 611.618384] Key type .llcrypt unregistered [ 611.624750] Key type ._llcrypt unregistered [ 617.358782] Key type ._llcrypt registered [ 617.363707] Key type .llcrypt registered [ 617.564286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.586023] alg: No test for adler32 (adler32-zlib) [ 618.526768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.617963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 618.652762] LNet: 29606:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 618.658914] LNet: 29606:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 618.664849] LNet: 29606:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 618.684213] LNet: Message forwarding will be enabled in 2s [ 621.793325] LNet: Message forwarding enabled [ 622.280272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.325262] LNet: Message forwarding will stop in 2s to 6s [ 625.891774] LNet: Message forwarding disabled [ 626.946144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.994370] LNet: 29704:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 627.000843] LNet: 29704:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 627.012064] LNet: 29704:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 627.049253] LNet: Message forwarding will be enabled in 2s [ 627.298366] LNet: 29718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.784554] Key type .llcrypt unregistered [ 628.790979] Key type ._llcrypt unregistered [ 645.705115] Key type ._llcrypt registered [ 645.706773] Key type .llcrypt registered [ 647.169623] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:36:41 (1788622601) [ 647.599272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.640576] alg: No test for adler32 (adler32-zlib) [ 648.594260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.705352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 655.597761] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:36:50 (1788622610) [ 655.773429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.790705] LNet: 31116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.845612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.943888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 656.000977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 656.048660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 656.127791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 656.193527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 656.277332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 656.338437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.385728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 656.445714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 656.490416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 656.535278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 656.620637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 656.707833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 656.748931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 656.804463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 656.862611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 656.925254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 656.989937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 657.028620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 657.045994] LNetError: 31161:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 664.528644] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:36:58 (1788622618) [ 664.638943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.672588] LNet: 31728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.719521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.827551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.438763] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:37:07 (1788622627) [ 674.591466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg446-server [ 676.757796] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:37:11 (1788622631) [ 676.950270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.991380] LNet: 32663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.029887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.116766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.143393] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 678.096284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[47-51]@tcp [ 678.115995] LNet: 32800:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 678.172486] LNetError: 32670:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 678.193323] LNetError: 32670:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.47@tcp has gone from up to down [ 685.387376] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:37:19 (1788622639) [ 687.229127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg446-server [ 689.202542] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:37:23 (1788622643) [ 689.374895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.403088] LNet: 33736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.411605] LNetError: 33736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.442486] LNet: Removed LNI 192.168.204.46@tcp [ 689.486263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.576963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.635604] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 690.538729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[47-52/2]@tcp [ 690.566310] LNet: 33854:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 690.575853] LNet: 33854:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 690.593613] LNetError: 33741:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 690.603293] LNetError: 33741:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 690.607800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[47-52/2]@tcp [ 690.610292] LNetError: 33741:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.47@tcp has gone from up to down [ 690.641302] LNetError: 33741:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 697.956861] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:37:32 (1788622652) [ 698.100964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.140356] LNet: 34444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.152398] LNetError: 34444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.174323] LNet: Removed LNI 192.168.204.46@tcp [ 698.195669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.276710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 698.352269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.422596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.535169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.617352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.663336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.715798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 698.797213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.833455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.959893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.076780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.121266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.179942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 699.280895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.364601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.464558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.566723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.611806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.684231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 699.777584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.814538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.864549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.937990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.987891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.054983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 700.138714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.194352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.312849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 700.446640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.500113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.568516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 700.645798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.711955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.767616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.174686] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:37:42 (1788622662) [ 710.294499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg446-server [ 712.695424] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:37:47 (1788622667) [ 715.059834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg446-server [ 717.141135] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:37:51 (1788622671) [ 717.418812] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.428225] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 718.432691] Key type .llcrypt unregistered [ 718.441778] Key type ._llcrypt unregistered [ 726.943855] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 727.119737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 727.123729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 727.128546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 730.212547] Key type ._llcrypt registered [ 730.215796] Key type .llcrypt registered [ 730.548651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.592642] alg: No test for adler32 (adler32-zlib) [ 731.516619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.595769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.630226] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 732.589515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.635936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 735.783474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 735.842337] LNet: Removed LNI 10.1.2.3@tcp [ 735.988301] LNet: 36564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.994914] LNetError: 36564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.009392] LNet: Removed LNI 192.168.204.46@tcp [ 736.888502] Key type .llcrypt unregistered [ 736.890440] Key type ._llcrypt unregistered [ 750.576822] Key type ._llcrypt registered [ 750.579306] Key type .llcrypt registered [ 751.941893] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:38:26 (1788622706) [ 752.625363] Key type .llcrypt unregistered [ 752.629460] Key type ._llcrypt unregistered [ 759.269175] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.336065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.340396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.343963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.439278] Key type ._llcrypt registered [ 762.443374] Key type .llcrypt registered [ 762.744501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.796272] alg: No test for adler32 (adler32-zlib) [ 763.770625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.867744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.932332] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 764.807982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.843891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.199189] LNet: 38054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.209627] LNetError: 38054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 768.215586] LNet: Removed LNI 10.1.2.3@tcp [ 768.242053] LNet: Removed LNI 192.168.204.46@tcp [ 769.091610] Key type .llcrypt unregistered [ 769.096586] Key type ._llcrypt unregistered [ 781.288383] Key type ._llcrypt registered [ 781.290317] Key type .llcrypt registered [ 782.764590] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:38:57 (1788622737) [ 783.462772] Key type .llcrypt unregistered [ 783.467134] Key type ._llcrypt unregistered [ 790.292470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.457791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.464875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.475497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.313528] Key type ._llcrypt registered [ 793.317920] Key type .llcrypt registered [ 793.603358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.636476] alg: No test for adler32 (adler32-zlib) [ 794.629429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.681509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 794.712682] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.369254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.391581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.661357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 798.691558] LNet: Removed LNI 10.1.2.3@tcp [ 798.708905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 798.757722] LNet: Removed LNI 10.1.2.31@tcp [ 798.773620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 798.797762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.462296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 799.487405] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 800.349546] LNet: 39969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.362368] LNetError: 39969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 800.377263] LNet: Removed LNI 10.1.2.31@tcp [ 800.391303] LNet: Removed LNI 10.1.2.3@tcp [ 800.978248] Key type .llcrypt unregistered [ 800.980492] Key type ._llcrypt unregistered [ 813.822591] Key type ._llcrypt registered [ 813.826257] Key type .llcrypt registered [ 815.314737] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:39:29 (1788622769) [ 815.857679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 815.877943] alg: No test for adler32 (adler32-zlib) [ 816.890317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.952453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.981882] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 817.853141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.890502] LNet: 40920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.902466] LNetError: 40920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.936536] LNet: Removed LNI 192.168.204.46@tcp [ 817.975786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.063870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.46@tcp [ 818.104582] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 818.950726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.46@tcp [ 818.979139] LNet: Removed LNI 192.168.204.46@tcp [ 819.027472] LNet: 41055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.692609] Key type .llcrypt unregistered [ 819.694743] Key type ._llcrypt unregistered [ 833.029355] Key type ._llcrypt registered [ 833.031781] Key type .llcrypt registered [ 834.422394] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:39:48 (1788622788) [ 837.004865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg446-server [ 839.094582] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:39:53 (1788622793) [ 839.786954] Key type .llcrypt unregistered [ 839.789213] Key type ._llcrypt unregistered [ 847.249822] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.586873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 847.593442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 847.602427] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 850.316291] Key type ._llcrypt registered [ 850.318637] Key type .llcrypt registered [ 850.522283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.548230] alg: No test for adler32 (adler32-zlib) [ 851.496873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.578644] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 852.309092] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 854.930844] LNet: 42886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.940467] LNetError: 42886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.946304] LNet: Removed LNI 192.168.204.46@tcp1 [ 854.957618] LNet: Removed LNI 10.1.2.3@tcp2 [ 855.682596] Key type .llcrypt unregistered [ 855.686076] Key type ._llcrypt unregistered [ 871.439408] Key type ._llcrypt registered [ 871.441803] Key type .llcrypt registered [ 873.137681] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:40:27 (1788622827) [ 873.647345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.674635] alg: No test for adler32 (adler32-zlib) [ 874.670358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.779784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.821807] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 875.864441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.886557] LNet: 43853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.892537] LNetError: 43853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.914308] LNet: Removed LNI 192.168.204.46@tcp [ 875.936612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.018680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 876.058600] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 883.551399] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:40:38 (1788622838) [ 883.690163] LNet: 44438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.696157] LNetError: 44438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.721197] LNet: Removed LNI 192.168.204.46@tcp [ 884.546953] Key type .llcrypt unregistered [ 884.549960] Key type ._llcrypt unregistered [ 892.089633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.345325] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.350790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.355028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.170775] Key type ._llcrypt registered [ 895.180789] Key type .llcrypt registered [ 895.498804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.526575] alg: No test for adler32 (adler32-zlib) [ 896.558846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.609701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.637549] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 897.663369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 897.701175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.197557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.223880] LNet: 45183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.232338] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.247802] LNet: Removed LNI 10.1.2.3@tcp [ 900.258049] LNet: Removed LNI 192.168.204.46@tcp [ 900.297379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.403074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 900.434862] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 900.443324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.047409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.078680] LNet: 45210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.087563] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.095342] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.120191] LNet: Removed LNI 10.1.2.3@tcp [ 901.139605] LNet: Removed LNI 192.168.204.46@tcp [ 901.164964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.234200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.272394] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 901.277411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.839506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 901.869458] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 902.875724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.896687] LNet: 45373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.903358] LNetError: 45373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.908820] LNetError: 45373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 902.914386] LNet: Removed LNI 10.1.2.3@tcp [ 902.916943] LNet: Removed LNI 192.168.204.46@tcp [ 902.929989] LNet: Removed LNI 192.168.204.46@tcp2 [ 902.946752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.068347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 903.112173] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 903.116952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.123482] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 912.706534] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:41:06 (1788622866) [ 912.934833] LNet: 45972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.945558] LNetError: 45972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.955301] LNetError: 45972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 912.974022] LNet: Removed LNI 10.1.2.3@tcp [ 912.980433] LNet: Removed LNI 192.168.204.46@tcp [ 913.010962] LNet: Removed LNI 192.168.204.46@tcp2 [ 913.807786] Key type .llcrypt unregistered [ 913.814875] Key type ._llcrypt unregistered [ 921.055674] Key type ._llcrypt registered [ 921.060704] Key type .llcrypt registered [ 921.334500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.362972] alg: No test for adler32 (adler32-zlib) [ 922.314670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.382885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.403272] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 923.463955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.484972] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.490723] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.507914] LNet: Removed LNI 192.168.204.46@tcp [ 923.533808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.605356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.631414] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 924.341359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.361606] LNet: 46395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.370917] LNetError: 46395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.392579] LNet: Removed LNI 192.168.204.46@tcp [ 924.461331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.562694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 924.595052] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 925.498423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.538320] LNet: 46425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.550206] LNetError: 46425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.575058] LNet: Removed LNI 192.168.204.46@tcp [ 925.620914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.722886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 925.790343] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 926.900125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 927.111595] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 927.121157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 927.130277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.851425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.898191] LNet: 46503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.906392] LNetError: 46503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.936202] LNet: Removed LNI 192.168.204.46@tcp [ 929.979707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.079677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.131491] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 938.625953] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:41:33 (1788622893) [ 939.011585] LNet: 47101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.023722] LNetError: 47101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.052905] LNet: Removed LNI 192.168.204.46@tcp [ 939.769404] Key type .llcrypt unregistered [ 939.773573] Key type ._llcrypt unregistered [ 946.808303] Key type ._llcrypt registered [ 946.818187] Key type .llcrypt registered [ 947.117542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.157610] alg: No test for adler32 (adler32-zlib) [ 948.235861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.345716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 948.418273] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 949.791476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.806561] LNet: 47497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.814246] LNetError: 47497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.839591] LNet: Removed LNI 192.168.204.46@tcp [ 949.860659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.936679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 949.973934] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 957.702727] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:41:52 (1788622912) [ 957.873278] LNet: 48082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.881293] LNetError: 48082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.921611] LNet: Removed LNI 192.168.204.46@tcp [ 958.662107] Key type .llcrypt unregistered [ 958.668219] Key type ._llcrypt unregistered [ 965.461105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.657430] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.688678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.698104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.417950] Key type ._llcrypt registered [ 968.424079] Key type .llcrypt registered [ 968.662693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.684591] alg: No test for adler32 (adler32-zlib) [ 969.625153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.710939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.760226] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 970.876336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.922420] LNet: 48537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.934711] LNetError: 48537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.962687] LNet: Removed LNI 192.168.204.46@tcp [ 970.991957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.091548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 971.126386] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 977.975373] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:42:12 (1788622932) [ 978.113856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.156947] LNet: 49132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.177213] LNetError: 49132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.220320] LNet: Removed LNI 192.168.204.46@tcp [ 978.252820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.351554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 985.481632] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:42:19 (1788622939) [ 985.593615] LNet: 49711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.709567] Key type .llcrypt unregistered [ 986.713975] Key type ._llcrypt unregistered [ 992.714864] Key type ._llcrypt registered [ 992.719066] Key type .llcrypt registered [ 992.949252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.993172] alg: No test for adler32 (adler32-zlib) [ 993.990906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.073155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.111394] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 995.650674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.674430] LNet: 50114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.684571] LNetError: 50114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.706250] LNet: Removed LNI 192.168.204.46@tcp [ 995.771482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.887855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 995.931799] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1003.127181] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:42:37 (1788622957) [ 1003.270093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.318321] LNet: 50703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.344729] LNetError: 50703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.366143] LNet: Removed LNI 192.168.204.46@tcp [ 1003.404678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.543983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1010.645592] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:42:45 (1788622965) [ 1012.337605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1014.300129] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:42:48 (1788622968) [ 1016.251205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1017.935629] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:42:52 (1788622972) [ 1019.633702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1021.868456] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:42:56 (1788622976) [ 1023.541766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1025.297291] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:42:59 (1788622979) [ 1026.765136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1028.700384] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:43:03 (1788622983) [ 1028.920274] LNet: 52758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.740763] Key type .llcrypt unregistered [ 1029.746590] Key type ._llcrypt unregistered [ 1037.265253] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1037.397432] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1037.403064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1037.409065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.044380] Key type ._llcrypt registered [ 1040.045946] Key type .llcrypt registered [ 1040.227908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.249498] alg: No test for adler32 (adler32-zlib) [ 1041.181756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.262539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.287242] LNet: Added LNI 192.168.204.46@tcp [8/512/0/180] [ 1041.296516] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1041.299995] LNet: 53074:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1041.407415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.436941] LNet: 53093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.441956] LNetError: 53093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.446653] LNet: Removed LNI 10.1.2.3@tcp [ 1041.451909] LNetError: 53093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.469342] LNet: Removed LNI 192.168.204.46@tcp [ 1041.515536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.614644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.643049] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1041.760933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.783601] LNet: 53117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.800532] LNetError: 53117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.815977] LNet: Removed LNI 192.168.204.46@tcp [ 1041.854567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.933838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.954066] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1041.957922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1041.960915] LNet: 53125:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.094587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.117300] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.124650] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.130728] LNet: Removed LNI 10.1.2.3@tcp [ 1042.133474] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.153904] LNet: Removed LNI 192.168.204.46@tcp [ 1042.192411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.269588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.288814] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1042.300165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.304917] LNet: 53151:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.519446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.550960] LNet: 53177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.558120] LNetError: 53177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.566996] LNet: Removed LNI 10.1.2.3@tcp [ 1042.569900] LNetError: 53177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.593992] LNet: Removed LNI 192.168.204.46@tcp [ 1042.656943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.758444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.789334] LNet: Added LNI 192.168.204.46@tcp [8/256/0/182] [ 1042.794914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1042.961381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.000569] LNet: 53205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.010380] LNetError: 53205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.021557] LNet: Removed LNI 10.1.2.3@tcp [ 1043.023213] LNetError: 53205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.044764] LNet: Removed LNI 192.168.204.46@tcp [ 1043.081150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.218210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.240611] LNet: Added LNI 192.168.204.46@tcp [8/256/0/182] [ 1043.251562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.255308] LNet: 53213:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.362719] LNet: Removed LNI 10.1.2.3@tcp [ 1043.377434] LNet: Removed LNI 192.168.204.46@tcp [ 1044.008837] Key type .llcrypt unregistered [ 1044.010554] Key type ._llcrypt unregistered [ 1055.879346] Key type ._llcrypt registered [ 1055.885954] Key type .llcrypt registered [ 1057.200394] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:43:31 (1788623011) [ 1058.888977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1060.708900] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:43:35 (1788623015) [ 1061.331128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.362340] alg: No test for adler32 (adler32-zlib) [ 1062.278726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.356343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.376434] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1063.061869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.46 [ 1063.083282] LNet: 54474:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1063.091145] LNet: Removed LNI 192.168.204.46@tcp [ 1069.591041] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:43:44 (1788623024) [ 1069.728894] LNet: 55038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.742473] LNetError: 55038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.764924] LNet: Removed LNI 192.168.204.46@tcp [ 1070.345548] Key type .llcrypt unregistered [ 1070.349603] Key type ._llcrypt unregistered [ 1077.355497] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.624704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.480073] Key type ._llcrypt registered [ 1080.483230] Key type .llcrypt registered [ 1080.713901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.731923] alg: No test for adler32 (adler32-zlib) [ 1082.128778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1089.754075] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:44:04 (1788623044) [ 1090.034905] LNet: 55954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.051216] LNetError: 55954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1090.074722] LNet: Removed LNI 10.1.2.3@tcp [ 1090.939538] Key type .llcrypt unregistered [ 1090.942555] Key type ._llcrypt unregistered [ 1098.234835] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1098.511786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1098.519089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1098.531524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.285945] Key type ._llcrypt registered [ 1101.290659] Key type .llcrypt registered [ 1101.490447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.508517] alg: No test for adler32 (adler32-zlib) [ 1102.526558] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1109.434825] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:44:23 (1788623063) [ 1111.406979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1113.458973] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:44:27 (1788623067) [ 1113.556188] LNet: 57127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.564940] LNetError: 57127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1113.588184] LNet: Removed LNI 10.1.2.3@tcp [ 1114.259477] Key type .llcrypt unregistered [ 1114.261988] Key type ._llcrypt unregistered [ 1120.927772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.206652] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.216645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.233099] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1123.973802] Key type ._llcrypt registered [ 1123.981851] Key type .llcrypt registered [ 1124.206729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.235575] alg: No test for adler32 (adler32-zlib) [ 1125.183349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.287956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1133.846281] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:44:48 (1788623088) [ 1134.243587] LNet: 58364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.249965] LNetError: 58364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1134.263695] LNet: Removed LNI 10.1.2.3@tcp [ 1134.950471] Key type .llcrypt unregistered [ 1134.952663] Key type ._llcrypt unregistered [ 1141.932905] Key type ._llcrypt registered [ 1141.937982] Key type .llcrypt registered [ 1142.243218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.258380] alg: No test for adler32 (adler32-zlib) [ 1143.352801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.383950] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1150.720613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1151.606775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1151.668392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1151.698444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1151.715768] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1151.993480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.033268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.072647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.122220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.181480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1152.244467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.289592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.337487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.398064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.418643] LNet: 58801:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.440225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.479594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.506920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.552477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.595974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.608649] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1152.611372] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1152.659058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.711185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1153.290497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1153.361988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1153.439983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1153.768515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.802831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.844888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.912511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.961544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1154.016780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.093531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.170741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.224224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.245462] LNet: 58912:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.288601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.373605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.444905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.502377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.581305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.638561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.704286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.460457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1155.527138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1155.575718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1155.597530] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1155.606263] LNet: Skipped 1 previous similar message [ 1155.878097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.920439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.962541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.035985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.119723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1156.159936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.190456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.224899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.285839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.311324] LNet: 59023:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.331266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.359746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.432550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.508902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.566680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.611083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.671262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1157.173849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1157.222699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1157.271769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1157.482148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.517503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.542596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.571563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.600341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1157.622700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.651865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.687461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.715462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.737437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.762840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.791172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.825403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.860639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.905235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.954186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.476603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1158.508452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1158.547461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1158.572296] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1158.579854] LNet: Skipped 1 previous similar message [ 1158.848931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.903969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.945631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.001100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.043826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1159.070768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.101684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.143475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.185677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.215786] LNet: 59246:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.223401] LNet: 59246:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1159.248318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.285561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.336082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.388968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.427394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.494749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.533414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.058293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1160.100247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1160.140420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1160.380359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.413326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.466896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.526701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.566708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1160.609652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.640156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.712399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.774294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.814359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.857557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.898096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.947890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.996471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.059514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.115685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.421356] LNet: 59398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.427929] LNetError: 59398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.463259] LNet: Removed LNI 192.168.204.46@tcp [ 1162.114569] Key type .llcrypt unregistered [ 1162.119557] Key type ._llcrypt unregistered [ 1179.537854] Key type ._llcrypt registered [ 1179.541429] Key type .llcrypt registered [ 1181.205832] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:45:35 (1788623135) [ 1182.165543] Key type .llcrypt unregistered [ 1182.172889] Key type ._llcrypt unregistered [ 1188.915297] Key type ._llcrypt registered [ 1188.919108] Key type .llcrypt registered [ 1189.249638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1189.278525] alg: No test for adler32 (adler32-zlib) [ 1190.210363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1190.241061] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1197.120525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1207.513384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.572214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.616739] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1209.053852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1209.081810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 1 [ 1209.126905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1209.162766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 1 [ 1209.192695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1209.227806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 1 [ 1209.305563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1209.375967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 1 [ 1209.439199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1209.478252] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1209.485578] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1209.733520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.790413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.833955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.892569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.935818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1209.983429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.019452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.062850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.105923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.148801] LNet: 60862:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.188511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.238478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.289790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.340129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.405972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.483545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.544497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1211.152916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1211.214567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 1 [ 1211.250914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1211.284201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 1 [ 1211.328875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1211.385531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 1 [ 1211.473805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1211.551679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 1 [ 1211.611594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1211.644790] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1211.650220] LNet: Skipped 2 previous similar messages [ 1211.896619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.946316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.003184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.047941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.095290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1212.157046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.225495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.251477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.297328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.315193] LNet: 60985:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.376453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.438259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.497276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.537430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.580893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.634293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.694859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.455690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1213.484703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 1 [ 1213.517216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1213.550892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 1 [ 1213.585358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1213.622397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 1 [ 1213.656441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1213.690742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 1 [ 1213.735368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1213.753358] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1213.757622] LNet: Skipped 2 previous similar messages [ 1214.020873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.048761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.075781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.103770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.132452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1214.171947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.201992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.245553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.284852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.315929] LNet: 61109:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.337157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.407272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.482614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.556975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.618744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.701138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.772396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.640333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1215.702185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 1 [ 1215.750887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1215.831600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 1 [ 1215.901598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1215.990701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 1 [ 1216.064713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1216.125367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 1 [ 1216.199172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1216.235080] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1216.241651] LNet: Skipped 2 previous similar messages [ 1216.582985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.612317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.635617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.666102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.699607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1216.732183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.790886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.881070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.925891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.943829] LNet: 61232:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.971348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1217.018695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.069317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.134159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.178625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.233714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.261070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.868735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1217.909901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 1 [ 1217.970149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1218.025522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 1 [ 1218.081651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1218.103206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 1 [ 1218.129920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1218.159287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 1 [ 1218.191835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1218.361185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.393205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.432095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.452520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.480564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1218.506324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.535980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.582633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.636668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.669548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.698170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.765904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.818569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.911844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.034758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.110556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.114103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1220.173102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_error -r 1 [ 1220.235942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1220.315454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_error -r 1 [ 1220.399777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1220.489109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_error -r 1 [ 1220.580729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1220.668959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_error -r 1 [ 1220.756468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1220.784512] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1220.790199] LNet: Skipped 5 previous similar messages [ 1221.292775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.385451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.473399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.541797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.624897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1221.694877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.812033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.926275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.037654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.091942] LNet: 61479:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.119619] LNet: 61479:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1222.153784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.198478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.276712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.316832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.357864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.383937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.413923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.847704] LNet: 61555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.859606] LNetError: 61555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.865197] LNet: Removed LNI 192.168.204.46@tcp [ 1228.909541] LNet: Removed LNI 192.168.204.46@tcp1 [ 1229.468834] Key type .llcrypt unregistered [ 1229.473065] Key type ._llcrypt unregistered [ 1244.595771] Key type ._llcrypt registered [ 1244.600888] Key type .llcrypt registered [ 1246.314289] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:46:40 (1788623200) [ 1247.410262] Key type .llcrypt unregistered [ 1247.412043] Key type ._llcrypt unregistered [ 1253.640527] Key type ._llcrypt registered [ 1253.643979] Key type .llcrypt registered [ 1253.919685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.930900] alg: No test for adler32 (adler32-zlib) [ 1254.916420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.977342] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1262.532480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1263.301929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1263.363192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1263.409448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1263.461028] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1263.656625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.686904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.722907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.764810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.797067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1263.835704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.882743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.917916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.962602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.984372] LNet: 62801:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.019301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.059666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.093431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.126279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.164535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.212642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.263771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.831306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1264.906702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1264.965150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1265.008377] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1265.291805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.340328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1265.417900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.474606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1265.533144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1265.578872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.618676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.671580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.725662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.752722] LNet: 62912:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.778059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.831342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.861839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.908905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.947866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.999726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.032652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1266.526818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1266.570791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1266.609862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1266.631945] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1266.952723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.982825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.011755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.046747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.086096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1267.119781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.157386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.216963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.270628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.300246] LNet: 63023:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1267.332554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.380133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.417516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.445404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.493321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.530541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.572680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.851807] LNet: 63064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.858952] LNetError: 63064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.883933] LNet: Removed LNI 192.168.204.46@tcp [ 1268.287652] Key type .llcrypt unregistered [ 1268.293702] Key type ._llcrypt unregistered [ 1286.530135] Key type ._llcrypt registered [ 1286.534231] Key type .llcrypt registered [ 1287.837302] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:47:22 (1788623242) [ 1288.738557] Key type .llcrypt unregistered [ 1288.745059] Key type ._llcrypt unregistered [ 1295.577700] Key type ._llcrypt registered [ 1295.580312] Key type .llcrypt registered [ 1295.951917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.968635] alg: No test for adler32 (adler32-zlib) [ 1296.911780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.941972] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1305.569092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1315.512283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.579881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.624662] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1317.100938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1317.137130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 1 [ 1317.186832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1317.231564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 1 [ 1317.299185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1317.341765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 1 [ 1317.376091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1317.426655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 1 [ 1317.480727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1317.514067] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1317.526769] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1317.798701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.876461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.954841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.022845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.064655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1318.108313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.177325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.228185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.292649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.313724] LNet: 64528:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.338315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.372595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.415871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.471594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.503847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.544755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.589902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1318.647108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.259660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1319.328742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_error -r 1 [ 1319.383611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1319.447360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_error -r 1 [ 1319.487373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1319.524676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_error -r 1 [ 1319.566391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1319.603747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_error -r 1 [ 1319.643265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1319.661865] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1319.669125] LNet: Skipped 2 previous similar messages [ 1319.919666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.972355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.031598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.079955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.121308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.157707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.179148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.202983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.236804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.275261] LNet: 64654:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.316727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.374850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.454810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.561918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.612838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.652850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.687128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1320.728268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.309296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1321.346182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_timeout -r 1 [ 1321.389285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1321.433714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_timeout -r 1 [ 1321.488593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1321.541479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_timeout -r 1 [ 1321.621782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1321.703864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_timeout -r 1 [ 1321.758573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1321.777554] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1321.787153] LNet: Skipped 1 previous similar message [ 1322.018386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.047613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1322.075688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1322.112283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1322.171703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1322.215375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1322.251885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1322.319910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.363664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.400696] LNet: 64779:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.431099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.465629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.511612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.544766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.575897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.603971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.637784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1322.667699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.167324] LNet: 64857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.179316] LNetError: 64857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.187577] LNet: Removed LNI 192.168.204.46@tcp [ 1328.200502] LNet: Removed LNI 192.168.204.46@tcp1 [ 1328.639546] Key type .llcrypt unregistered [ 1328.646204] Key type ._llcrypt unregistered [ 1346.210911] Key type ._llcrypt registered [ 1346.219145] Key type .llcrypt registered [ 1347.341950] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:48:22 (1788623302) [ 1349.148289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1350.824389] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:48:25 (1788623305) [ 1351.873782] Key type .llcrypt unregistered [ 1351.880225] Key type ._llcrypt unregistered [ 1358.751758] Key type ._llcrypt registered [ 1358.753268] Key type .llcrypt registered [ 1358.944660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1358.960696] alg: No test for adler32 (adler32-zlib) [ 1359.861399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1359.901403] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1368.463457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1369.476601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1369.537963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1369.600924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1369.631744] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1369.978056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.031952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1370.089621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.141752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.143641] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1370.151444] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1370.212682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1370.274972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.390650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.447400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.488314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.519527] LNet: 66411:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.556824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.610787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.661980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.723622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.796624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.869229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1371.241382] LNet: 66450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.245358] LNetError: 66450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.262406] LNet: Removed LNI 192.168.204.46@tcp [ 1372.839639] Key type .llcrypt unregistered [ 1372.843462] Key type ._llcrypt unregistered [ 1392.564678] Key type ._llcrypt registered [ 1392.568175] Key type .llcrypt registered [ 1392.830714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1392.855950] alg: No test for adler32 (adler32-zlib) [ 1393.873110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1393.916283] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1402.168596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1411.812044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.866847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.894814] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1413.752863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1413.786307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e network_timeout -r 1 [ 1413.823140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1413.874577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e network_timeout -r 1 [ 1413.909977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1413.957916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e network_timeout -r 1 [ 1414.001658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1414.056928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e network_timeout -r 1 [ 1414.097264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1414.120234] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1414.376576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1414.399289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1414.424197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1414.431183] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1414.437301] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1414.486291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1414.511509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1414.541861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1414.566086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.605726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.642633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.682755] LNet: 67333:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.723426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1414.749895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1414.785289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.831429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1414.864912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.896747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1419.563293] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1420.055602] LNet: 67407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.074514] LNetError: 67407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.088068] LNet: Removed LNI 192.168.204.46@tcp [ 1420.113675] LNet: Removed LNI 192.168.204.46@tcp1 [ 1420.736143] Key type .llcrypt unregistered [ 1420.739235] Key type ._llcrypt unregistered [ 1439.554420] Key type ._llcrypt registered [ 1439.557312] Key type .llcrypt registered [ 1441.301279] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:49:55 (1788623395) [ 1442.007409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.046904] alg: No test for adler32 (adler32-zlib) [ 1443.050953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.133941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1443.158909] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1444.351948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1444.379205] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1445.495926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1445.555713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1445.601500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1445.659845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1445.713758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1445.790281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1445.847584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1445.863577] LNetError: 68504:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1446.175296] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1446.178712] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1448.223591] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1448.749949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1452.319734] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1452.995365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1453.047300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.159596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.207122] LNet: 68601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.220113] LNetError: 68601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.244106] LNet: Removed LNI 192.168.204.46@tcp [ 1453.270873] LNet: Removed LNI 192.168.204.46@tcp1 [ 1453.359842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.527484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.569581] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1454.586540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.645909] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1455.988424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1456.042474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.118442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1456.183186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1456.250028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1456.278579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1456.328535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1456.366234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1456.379674] LNetError: 68874:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1456.543447] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1458.820531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1462.693237] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1462.704125] LNet: Skipped 1 previous similar message [ 1467.428917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1467.462843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1467.506822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.474872] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:50:30 (1788623430) [ 1476.545936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.562176] LNet: 69573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.572152] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.579686] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1476.591762] LNet: Removed LNI 192.168.204.46@tcp [ 1476.610683] LNet: Removed LNI 192.168.204.46@tcp1 [ 1476.632628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.695525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.737447] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1477.514658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1477.555901] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1478.425535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1478.550349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1478.600352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1478.659863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1478.722277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1478.777241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1478.817233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 0 [ 1478.998041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1479.776103] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.46@tcp [ 1479.776123] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1479.800938] LNet: Skipped 1 previous similar message [ 1493.903765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1494.099793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1494.170153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1494.247984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 500 [ 1494.424833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.249647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.319947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.337126] LNet: 70065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.342642] LNetError: 70065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.348307] LNetError: 70065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.354607] LNet: Removed LNI 192.168.204.46@tcp [ 1501.372480] LNet: Removed LNI 192.168.204.46@tcp1 [ 1501.397778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.452147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.485085] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1502.310509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.342383] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1503.349058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1503.403083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1503.469158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1503.509658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1503.549757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1503.589473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1503.628985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.690402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 0 [ 1504.481870] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1504.492159] LNet: Skipped 3 previous similar messages [ 1505.106652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1515.427867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1515.464092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1515.508536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1523.166657] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:51:17 (1788623477) [ 1523.693643] LNet: 71114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.701488] LNetError: 71114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.709566] LNetError: 71114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1523.720341] LNet: Removed LNI 192.168.204.46@tcp [ 1523.731539] LNet: Removed LNI 192.168.204.46@tcp1 [ 1524.707203] Key type .llcrypt unregistered [ 1524.710456] Key type ._llcrypt unregistered [ 1533.697739] Key type ._llcrypt registered [ 1533.711783] Key type .llcrypt registered [ 1534.197481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1534.243661] alg: No test for adler32 (adler32-zlib) [ 1535.198286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1535.253784] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1546.791671] Lustre: DEBUG MARKER: Initial discovery [ 1546.870072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1550.289360] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1550.334677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1550.443308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1550.471471] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1550.519990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1550.582382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.146@tcp -r 1 -m GET -l 3 [ 1550.623456] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1550.631035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.146@tcp -r 1 -m GET -e local_error [ 1550.711160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.146@tcp -r 1 -m PUT -l 6 [ 1552.391914] Lustre: DEBUG MARKER: Force oleg446-client.virtnet to discover 192.168.204.146@tcp (in background) [ 1552.523247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1554.702307] Lustre: DEBUG MARKER: Force oleg446-server to discover 192.168.204.46@tcp [ 1555.617696] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1557.727160] Lustre: DEBUG MARKER: Wait for 71868 [ 1558.690608] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1559.683368] Lustre: DEBUG MARKER: Finished wait on 71868 [ 1559.853260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1559.998909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1560.300285] LNet: 72325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.306415] LNetError: 72325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.325689] LNet: Removed LNI 192.168.204.46@tcp [ 1560.695564] Key type .llcrypt unregistered [ 1560.700592] Key type ._llcrypt unregistered [ 1579.023914] Key type ._llcrypt registered [ 1579.028223] Key type .llcrypt registered [ 1580.597684] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:52:14 (1788623534) [ 1581.401908] Key type .llcrypt unregistered [ 1581.406795] Key type ._llcrypt unregistered [ 1589.056138] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1589.320205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1589.327819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1589.333892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1592.213122] Key type ._llcrypt registered [ 1592.215466] Key type .llcrypt registered [ 1592.527669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1592.556592] alg: No test for adler32 (adler32-zlib) [ 1593.537043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.636599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.692028] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1594.551545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1594.580605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1604.383698] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:52:38 (1788623558) [ 1604.556283] LNet: 74399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.568959] LNetError: 74399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1604.582463] LNet: Removed LNI 10.1.2.3@tcp [ 1604.606603] LNet: Removed LNI 192.168.204.46@tcp [ 1605.333759] Key type .llcrypt unregistered [ 1605.338487] Key type ._llcrypt unregistered [ 1613.269204] Key type ._llcrypt registered [ 1613.271816] Key type .llcrypt registered [ 1613.525992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.540477] alg: No test for adler32 (adler32-zlib) [ 1614.515655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.580274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.608424] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1615.352875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1615.369151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1616.803890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.46@tcp [ 1617.887748] LNetError: Unexpected error -22 connecting to 192.168.204.46@tcp at host 192.168.204.46:988 [ 1624.784123] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:52:59 (1788623579) [ 1624.881951] LNet: 75524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.891461] LNetError: 75524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1624.905837] LNet: Removed LNI 10.1.2.3@tcp [ 1624.922157] LNet: Removed LNI 192.168.204.46@tcp [ 1625.709731] Key type .llcrypt unregistered [ 1625.713828] Key type ._llcrypt unregistered [ 1632.566186] Key type ._llcrypt registered [ 1632.568221] Key type .llcrypt registered [ 1632.760856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1632.799615] alg: No test for adler32 (adler32-zlib) [ 1633.740513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.793529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1633.825414] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1634.533395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1634.561855] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 1635.275790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.46@tcp1 --nid 192.168.204.46@tcp2 [ 1639.608813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp1 192.168.204.46@tcp2 [ 1639.634503] LNetError: 76364:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.46@tcp1 to 192.168.204.46@tcp2 - Route table may be misconfigured [ 1639.672802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp2 192.168.204.46@tcp1 [ 1645.687338] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:53:20 (1788623600) [ 1645.826738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.858856] LNet: 76932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.870504] LNetError: 76932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.881901] LNet: Removed LNI 192.168.204.46@tcp1 [ 1646.950359] LNet: Removed LNI 192.168.204.46@tcp2 [ 1646.981627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.035280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.073980] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1647.749536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.767283] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1648.580710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1648.619316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.46@tcp [ 1648.655784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp1 192.168.204.46@tcp1 [ 1648.690395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.46@tcp -d 192.168.204.46@tcp -e network_timeout [ 1648.726373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -e network_timeout [ 1648.760881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -e network_timeout [ 1648.793976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -e network_timeout [ 1648.844909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1648.871954] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1648.892649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1649.010949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.057242] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1649.105722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1655.795079] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:53:30 (1788623610) [ 1655.884264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.908204] LNet: 77793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.930675] LNetError: 77793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.937732] LNetError: 77793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1655.947711] LNet: Removed LNI 192.168.204.46@tcp [ 1656.999425] LNet: Removed LNI 192.168.204.46@tcp1 [ 1657.050582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.212822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1657.588787] LNet: 77838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.016534] Key type .llcrypt unregistered [ 1658.027422] Key type ._llcrypt unregistered [ 1669.073303] Key type ._llcrypt registered [ 1669.076554] Key type .llcrypt registered [ 1670.600162] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:53:44 (1788623624) [ 1671.545377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.573733] alg: No test for adler32 (adler32-zlib) [ 1672.510343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.894888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1676.375611] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:53:50 (1788623630) [ 1676.486250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.537401] LNet: 78926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.589702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.713268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.743772] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1677.901438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1677.920977] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1679.366359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1679.416896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp1 [ 1679.469300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp1 [ 1686.870777] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:54:01 (1788623641) [ 1688.700844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg446-server [ 1691.125319] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:54:05 (1788623645) [ 1693.615573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg446-server [ 1695.970408] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:54:10 (1788623650) [ 1698.867262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg446-server [ 1701.274455] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:54:15 (1788623655) [ 1703.616940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg446-server [ 1705.190815] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:54:20 (1788623660) [ 1707.103440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg446-server [ 1709.470571] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:54:23 (1788623663) [ 1711.637970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg446-server [ 1713.704946] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:54:28 (1788623668) [ 1715.650670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg446-server [ 1717.538902] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:54:32 (1788623672) [ 1719.206561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg446-server [ 1720.877606] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:54:35 (1788623675) [ 1722.644603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg446-server [ 1724.434465] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:54:39 (1788623679) [ 1724.655544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.692259] LNet: 82812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.699693] LNetError: 82812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.708306] LNet: Removed LNI 192.168.204.46@tcp [ 1725.740832] LNet: Removed LNI 192.168.204.46@tcp1 [ 1725.770656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.858116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.914892] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1726.922216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1727.012809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1727.255757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.307674] LNet: 82959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.318868] LNetError: 82959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.330384] LNetError: 82959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1727.364827] LNet: Removed LNI 192.168.204.46@tcp [ 1727.414413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.504780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1727.549149] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1728.794489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1729.095232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.118954] LNetError: 83104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.159578] LNet: Removed LNI 192.168.204.46@tcp [ 1730.230220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.355537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.410217] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1731.603233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1731.707250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1731.925497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.945870] LNet: 83252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.949539] LNet: 83252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1731.954694] LNetError: 83252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.029065] LNet: Removed LNI 192.168.204.46@tcp [ 1733.105507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.221650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1733.264811] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1734.625682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1735.004977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.102776] LNet: Removed LNI 192.168.204.46@tcp [ 1736.166915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.302448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.346055] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1737.333176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1737.423396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1737.781951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.817743] LNet: 83545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.821789] LNet: 83545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1737.850732] LNetError: 83545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.867264] LNetError: 83545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1737.888990] LNet: Removed LNI 192.168.204.46@tcp [ 1737.932576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.024053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1738.051966] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1739.075957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1739.389458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.467234] LNet: Removed LNI 192.168.204.46@tcp [ 1740.549156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.672583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.732580] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1741.559301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1741.633148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1741.936518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.083162] LNet: Removed LNI 192.168.204.46@tcp [ 1743.144275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.216877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1743.251437] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1744.376426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1744.687865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.791876] LNet: Removed LNI 192.168.204.46@tcp [ 1745.851694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.969907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.040371] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1747.242862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1747.331644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1747.690296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.723469] LNet: 84130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.730655] LNet: 84130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1747.747244] LNetError: 84130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.756632] LNetError: 84130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1747.776163] LNet: Removed LNI 192.168.204.46@tcp [ 1747.811816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.899564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1747.941349] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1749.102524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1749.439686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.507488] LNet: Removed LNI 192.168.204.46@tcp [ 1750.561873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.674433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.751817] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1751.827621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1751.894664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1752.255214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.325887] LNet: Removed LNI 192.168.204.46@tcp [ 1753.373298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.468825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1753.503740] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1754.316369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1754.643887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.693635] LNet: Removed LNI 192.168.204.46@tcp [ 1755.739867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.865892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.905709] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1757.096657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1757.172597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.214358] LNet: Removed LNI 192.168.204.46@tcp [ 1757.250985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.345061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.378080] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1758.240387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1758.327966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.383210] LNet: Removed LNI 192.168.204.46@tcp [ 1758.406938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.457191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1758.526199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.583579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.659266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1758.745352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.794331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.860056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1758.949583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.999728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.083142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1759.114675] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1759.923064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.957405] LNet: Removed LNI 192.168.204.46@tcp [ 1759.974256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.036934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1768.276807] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:55:22 (1788623722) [ 1768.381152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.419392] LNet: 85583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.429632] LNet: 85583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1768.451380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.560797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1768.584624] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1769.803791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.825550] LNetError: 85728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.837809] LNetError: 85728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1769.860369] LNet: Removed LNI 192.168.204.46@tcp231 [ 1769.900722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.981286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1770.010873] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1770.019433] LNet: 85736:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1770.525330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1770.564969] LNet: Removed LNI 192.168.204.46@tcp231 [ 1770.606935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1770.657198] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1771.838419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.872507] LNet: Removed LNI 192.168.204.46@tcp231 [ 1771.901262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.999574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.053773] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/0] [ 1772.066779] LNet: 85888:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.475203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1772.525083] LNet: Removed LNI 192.168.204.46@tcp231 [ 1772.546714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1772.587917] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/0] [ 1773.832626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.870833] LNet: Removed LNI 192.168.204.46@tcp231 [ 1773.892326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.976655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1774.020115] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/60] [ 1774.028838] LNet: 86039:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.647770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1774.677684] LNet: Removed LNI 192.168.204.46@tcp231 [ 1774.739368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1774.793133] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/60] [ 1776.094984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.142361] LNet: Removed LNI 192.168.204.46@tcp231 [ 1776.174285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.290725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1776.326945] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1776.345798] LNet: 86191:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.882440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1776.938211] LNet: Removed LNI 192.168.204.46@tcp231 [ 1776.970220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1777.006043] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1785.453537] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:55:40 (1788623740) [ 1785.575671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.636682] LNet: Removed LNI 192.168.204.46@tcp231 [ 1785.650594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.701887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1785.722577] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1786.345413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1786.394778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1786.432457] LNet: Removed LNI 192.168.204.46@tcp232 [ 1786.463337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1786.487180] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1787.176520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.222409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.255044] LNet: Removed LNI 192.168.204.46@tcp232 [ 1787.282630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1787.328644] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1788.270919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1788.319360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1788.364048] LNet: Removed LNI 192.168.204.46@tcp232 [ 1788.405923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1788.452096] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1789.376788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1789.428866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1789.475862] LNet: Removed LNI 192.168.204.46@tcp232 [ 1789.505586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1797.220917] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:55:51 (1788623751) [ 1798.617603] Key type .llcrypt unregistered [ 1798.621433] Key type ._llcrypt unregistered [ 1806.465159] Key type ._llcrypt registered [ 1806.466984] Key type .llcrypt registered [ 1806.665892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1806.701910] alg: No test for adler32 (adler32-zlib) [ 1807.811860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1807.865770] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1816.313991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1826.367957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.424850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1826.452286] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1827.947362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 2 [ 1827.995467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 2 [ 1828.044960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 2 [ 1828.094727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 2 [ 1828.135990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_interrupt -r 2 [ 1828.176367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 2 [ 1828.224325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_interrupt -r 2 [ 1828.287158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 2 [ 1828.348095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1828.374073] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1828.381234] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1828.815504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.850763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1828.884490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.920351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1828.951765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1828.984293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.025771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.061680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.095942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.114041] LNet: 88706:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.129269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.158390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.198925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.237257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.290793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.321246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1829.874753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 2 [ 1829.928681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 2 [ 1830.029534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 2 [ 1830.074599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 2 [ 1830.121603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_dropped -r 2 [ 1830.160900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 2 [ 1830.240490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_dropped -r 2 [ 1830.315526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 2 [ 1830.374770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1830.438647] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1830.877866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1830.929737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1830.964786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.000608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.065580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1831.113501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.156665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.196772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.232351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.249655] LNet: 88829:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.276584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.318664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.363855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.404463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.461518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.510647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.185355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 2 [ 1832.240660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 2 [ 1832.277683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 2 [ 1832.358578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 2 [ 1832.406882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_aborted -r 2 [ 1832.446607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 2 [ 1832.491678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_aborted -r 2 [ 1832.541740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 2 [ 1832.587320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1832.615865] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1832.921773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1832.962722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1832.989208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.025423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.069829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1833.147631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.199866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.263752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.314690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.363280] LNet: 88951:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.422605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.492065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.540667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.606158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.659319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.710157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1834.448975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 2 [ 1834.514781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 2 [ 1834.619812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 2 [ 1834.694802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 2 [ 1834.776059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_no_route -r 2 [ 1834.854472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 2 [ 1834.910698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_no_route -r 2 [ 1834.966624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 2 [ 1835.036520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1835.067962] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1835.558580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.590882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1835.624722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1835.654603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1835.697978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1835.732761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1835.765390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1835.795368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.828895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1835.850036] LNet: 89074:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1835.873571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1835.903075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1835.936373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1835.963868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1835.995920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.033719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1837.011328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 2 [ 1837.078585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 2 [ 1837.180782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 2 [ 1837.236993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 2 [ 1837.300600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_timeout -r 2 [ 1837.354911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 2 [ 1837.407866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_timeout -r 2 [ 1837.458770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 2 [ 1837.488890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1837.945322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.017560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1838.085529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.127693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1838.202344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1838.250437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1838.312343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1838.394091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1838.467312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1838.537405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1838.602472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1838.658637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.747633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.785170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.857319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1839.620134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 2 [ 1839.650167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 2 [ 1839.682705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 2 [ 1839.727549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 2 [ 1839.802802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_dropped -r 2 [ 1839.844599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 2 [ 1839.892884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_dropped -r 2 [ 1839.978044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 2 [ 1840.041400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1840.088155] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1840.094483] LNet: Skipped 1 previous similar message [ 1840.102041] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1840.421140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.453140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1840.484179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1840.520720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1840.555499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1840.607182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1840.641463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1840.719442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.761715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.779622] LNet: 89317:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1840.785560] LNet: 89317:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1840.802805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.843216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.881626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.928757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.972486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.999014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1846.115461] LNet: 89392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.129523] LNetError: 89392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.135409] LNet: Removed LNI 192.168.204.46@tcp [ 1847.142770] LNetError: 89392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.174994] LNet: Removed LNI 192.168.204.46@tcp1 [ 1847.716191] Key type .llcrypt unregistered [ 1847.719852] Key type ._llcrypt unregistered [ 1868.289364] Key type ._llcrypt registered [ 1868.290850] Key type .llcrypt registered [ 1869.902992] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:57:04 (1788623824) [ 1870.404833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.421179] alg: No test for adler32 (adler32-zlib) [ 1871.369812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.416526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.437241] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1872.636038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1872.692081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1872.727217] LNetError: 90370:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1872.877528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.46@tcp [ 1873.017177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1880.214835] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:57:14 (1788623834) [ 1880.557823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.613740] LNet: 90960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.624920] LNetError: 90960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.646252] LNet: Removed LNI 192.168.204.46@tcp [ 1880.684389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.800864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.859950] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1881.952516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1881.988854] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1883.376385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.46@tcp [ 1883.415719] LNetError: 91226:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1883.810210] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1883.815501] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1884.778402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.46@tcp [ 1894.119357] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:57:28 (1788623848) [ 1896.975433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg446-server [ 1899.898558] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:57:33 (1788623853) [ 1902.382769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg446-server [ 1905.049353] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:57:39 (1788623859) [ 1907.695927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg446-server [ 1910.024360] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:57:44 (1788623864) [ 1910.300329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.336405] LNet: 92849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.343736] LNetError: 92849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.391266] LNet: Removed LNI 192.168.204.46@tcp [ 1911.396373] LNetError: 92849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.408478] LNet: Removed LNI 192.168.204.46@tcp1 [ 1911.437437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.545087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.585916] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1912.505802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1912.616030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1912.667707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1912.853514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1912.924425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1913.038494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1921.717523] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:57:56 (1788623876) [ 1921.982964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.018136] LNet: 93581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.031941] LNetError: 93581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.050875] LNet: Removed LNI 192.168.204.46@tcp [ 1922.093992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.170747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.213696] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1929.461993] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:58:04 (1788623884) [ 1929.612174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.651776] LNet: 94286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.664844] LNetError: 94286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.679945] LNet: Removed LNI 192.168.204.46@tcp [ 1929.692910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.756647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.789313] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1936.677149] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:58:11 (1788623891) [ 1938.251817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1940.419802] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:58:14 (1788623894) [ 1941.089294] LNet: 95321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.093670] LNetError: 95321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.124284] LNet: Removed LNI 192.168.204.46@tcp [ 1941.694939] Key type .llcrypt unregistered [ 1941.698280] Key type ._llcrypt unregistered [ 1948.436832] Key type ._llcrypt registered [ 1948.443935] Key type .llcrypt registered [ 1948.750745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1948.770888] alg: No test for adler32 (adler32-zlib) [ 1949.705930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1949.734276] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1957.845061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1963.364864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.146@tcp [ 1963.876819] LNet: 95737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.885525] LNetError: 95737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.896276] LNet: Removed LNI 192.168.204.46@tcp [ 1964.477821] Key type .llcrypt unregistered [ 1964.488154] Key type ._llcrypt unregistered [ 1978.799648] Key type ._llcrypt registered [ 1978.801889] Key type .llcrypt registered [ 1980.358775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1982.140748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1985.105454] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:58:59 (1788623939) [ 1987.109941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1989.314816] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:59:03 (1788623943) [ 1991.772591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg446-server [ 1994.279472] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:59:08 (1788623948) [ 1996.478326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg446-server [ 1999.438511] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:59:13 (1788623953) [ 2001.604255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.657611] alg: No test for adler32 (adler32-zlib) [ 2002.768710] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2003.127996] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2004.856043] Key type lgssc registered [ 2006.348186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.257266] Key type lgssc unregistered [ 2030.658574] LNet: 98911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.668225] LNetError: 98911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.688207] LNet: Removed LNI 192.168.204.46@tcp [ 2031.498548] Key type .llcrypt unregistered [ 2031.503597] Key type ._llcrypt unregistered [ 2048.780342] Key type ._llcrypt registered [ 2048.782909] Key type .llcrypt registered [ 2050.159319] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:00:04 (1788624004) [ 2050.773703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.790274] alg: No test for adler32 (adler32-zlib) [ 2051.768213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.830406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2051.895667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2051.937912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2051.995409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2059.460648] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:00:13 (1788624013) [ 2059.644834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.711481] LNet: 100318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.769843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.899549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.961481] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2061.088570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.118207] LNet: 100468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.123862] LNetError: 100468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.139849] LNet: Removed LNI 192.168.204.46@tcp [ 2061.170165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.250593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2061.277441] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2061.751957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.773494] LNetError: 100496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.793231] LNet: Removed LNI 192.168.204.46@tcp [ 2061.814687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.876772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2061.917743] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2070.001505] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:00:24 (1788624024) [ 2070.115270] LNet: 101078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.119037] LNet: 101078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2070.127174] LNetError: 101078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.141780] LNet: Removed LNI 192.168.204.46@tcp [ 2071.056631] Key type .llcrypt unregistered [ 2071.060433] Key type ._llcrypt unregistered [ 2079.291563] Key type ._llcrypt registered [ 2079.293220] Key type .llcrypt registered [ 2079.701955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.734584] alg: No test for adler32 (adler32-zlib) [ 2080.725461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.879542] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2080.886963] LNet: *** cfs_fail_loc=e003, val=100*** [ 2081.996563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2082.016258] LNet: *** cfs_fail_loc=e003, val=100*** [ 2082.018510] LNet: Skipped 2 previous similar messages [ 2082.025890] Ping reply received with mapping: [ 2082.028697] found metadata reply NI[192.168.204.46@tcp] [ 2082.031534] 2e cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2082.036950] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2082.040525] 5a 5a 5a 5a 5a 5a 5a 5a [ 2082.200435] LNet: 101482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.208851] LNetError: 101482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.238350] LNet: Removed LNI 192.168.204.46@tcp [ 2084.147702] Key type .llcrypt unregistered [ 2084.152801] Key type ._llcrypt unregistered [ 2097.407031] Key type ._llcrypt registered [ 2097.409821] Key type .llcrypt registered [ 2099.170384] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:00:53 (1788624053) [ 2099.905408] Key type .llcrypt unregistered [ 2099.910037] Key type ._llcrypt unregistered [ 2108.276677] Key type ._llcrypt registered [ 2108.280147] Key type .llcrypt registered [ 2108.619781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2108.639553] alg: No test for adler32 (adler32-zlib) [ 2109.917693] LNetError: 102533:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2110.372854] Key type .llcrypt unregistered [ 2110.378305] Key type ._llcrypt unregistered [ 2116.392928] Key type ._llcrypt registered [ 2116.396310] Key type .llcrypt registered [ 2117.915986] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:01:12 (1788624072) [ 2118.664293] Key type .llcrypt unregistered [ 2118.667111] Key type ._llcrypt unregistered [ 2126.790874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2126.801075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2129.353189] Key type ._llcrypt registered [ 2129.356986] Key type .llcrypt registered [ 2129.584468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2129.599436] alg: No test for adler32 (adler32-zlib) [ 2130.522850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.590590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2141.910393] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:01:36 (1788624096) [ 2142.194288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.238694] LNet: 104309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.255806] LNetError: 104309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2142.285164] LNet: Removed LNI 10.1.2.3@tcp [ 2142.317340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.180930] Key type .llcrypt unregistered [ 2143.184639] Key type ._llcrypt unregistered [ 2150.801091] Key type ._llcrypt registered [ 2150.803397] Key type .llcrypt registered [ 2151.126249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.167561] alg: No test for adler32 (adler32-zlib) [ 2152.218153] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2152.309822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2152.539492] LNet: 104595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.557580] LNetError: 104595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.588394] LNet: Removed LNI 192.168.204.46@tcp [ 2153.585645] Key type .llcrypt unregistered [ 2153.587226] Key type ._llcrypt unregistered [ 2166.849678] Key type ._llcrypt registered [ 2166.853778] Key type .llcrypt registered [ 2168.842616] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:02:02 (1788624122) [ 2169.736993] Key type .llcrypt unregistered [ 2169.739389] Key type ._llcrypt unregistered [ 2177.642780] Key type ._llcrypt registered [ 2177.645130] Key type .llcrypt registered [ 2178.031461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2178.057373] alg: No test for adler32 (adler32-zlib) [ 2186.894898] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:02:21 (1788624141) [ 2187.501679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.560407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.648087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.704913] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2188.608292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2188.640147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2188.672895] LNet: Removed LNI 192.168.204.46@tcp [ 2188.948468] LNet: 106352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.516319] Key type .llcrypt unregistered [ 2189.518332] Key type ._llcrypt unregistered [ 2203.309401] Key type ._llcrypt registered [ 2203.313682] Key type .llcrypt registered [ 2204.987762] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:02:39 (1788624159) [ 2205.846968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.886560] alg: No test for adler32 (adler32-zlib) [ 2206.869923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.931817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.947601] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2207.999842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2208.089900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.46@tcp [ 2208.128960] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2208.139593] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2208.146645] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2208, 0, 0 [ 2208.158252] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2208.162940] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2208.173078] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2208.178123] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2208, 0, 0 [ 2208.190387] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2208.195580] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2208.203087] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2208.209965] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2208, 0, 2258 [ 2208.227572] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2208.241758] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2208.253740] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2208.259692] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2208, 0, 2258 [ 2208.276293] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2215.995640] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:02:50 (1788624170) [ 2216.663606] LNet: 107890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.673827] LNetError: 107890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.692632] LNet: Removed LNI 192.168.204.46@tcp [ 2217.380611] Key type .llcrypt unregistered [ 2217.382345] Key type ._llcrypt unregistered [ 2226.577422] Key type ._llcrypt registered [ 2226.579662] Key type .llcrypt registered [ 2226.835493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2226.870412] alg: No test for adler32 (adler32-zlib) [ 2227.847532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2227.901953] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2238.154911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2248.841782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.908129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2248.962294] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 2250.415354] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2250.698878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2250.703186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2250.707550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2253.115375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2253.149497] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2265.202231] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:03:40 (1788624220) [ 2265.556232] LNet: 109398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.565495] LNetError: 109398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.574279] LNet: Removed LNI 192.168.204.46@tcp [ 2266.591277] LNet: Removed LNI 192.168.204.46@tcp1 [ 2266.593278] LNetError: 109398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2266.609985] LNetError: 109398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2266.651143] LNet: Removed LNI 10.1.2.3@tcp99 [ 2267.314125] Key type .llcrypt unregistered [ 2267.318242] Key type ._llcrypt unregistered [ 2275.056618] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2275.069016] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2278.038279] Key type ._llcrypt registered [ 2278.040904] Key type .llcrypt registered [ 2278.345920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2278.367963] alg: No test for adler32 (adler32-zlib) [ 2279.403634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.475766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.519710] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2280.245774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2280.262644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2282.655920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2282.705947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2282.743385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2282.764980] LNetError: 110141:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2282.786414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2282.842957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2283.305799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2283.404199] LNet: 110161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.411690] LNetError: 110161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2283.416919] LNet: Removed LNI 10.1.2.3@tcp [ 2284.453913] LNet: Removed LNI 192.168.204.46@tcp [ 2285.305347] Key type .llcrypt unregistered [ 2285.307850] Key type ._llcrypt unregistered [ 2297.702096] Key type ._llcrypt registered [ 2297.705277] Key type .llcrypt registered [ 2299.592815] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:04:13 (1788624253) [ 2300.315102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.333932] alg: No test for adler32 (adler32-zlib) [ 2301.402746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.530165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.578857] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2302.381789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2302.469262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg446-client@tcp [ 2311.015933] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:04:25 (1788624265) [ 2311.274477] LNet: 111692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.305509] LNetError: 111692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.390294] LNet: Removed LNI 192.168.204.46@tcp [ 2312.237504] Key type .llcrypt unregistered [ 2312.243344] Key type ._llcrypt unregistered [ 2320.487846] Key type ._llcrypt registered [ 2320.491339] Key type .llcrypt registered [ 2320.801980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.825342] alg: No test for adler32 (adler32-zlib) [ 2321.720510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2321.755409] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2321.785692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2329.200563] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:04:43 (1788624283) [ 2331.037622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2333.104915] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:04:47 (1788624287) [ 2334.762804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2336.876562] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:04:51 (1788624291) [ 2337.189238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.206532] LNet: 113115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.212214] LNetError: 113115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.229389] LNet: Removed LNI 192.168.204.46@tcp [ 2337.247443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.311821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.334375] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2338.066673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2338.111363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2346.155870] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:05:00 (1788624300) [ 2346.572238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.602158] LNet: 113825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.610666] LNetError: 113825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.636641] LNet: Removed LNI 192.168.204.46@tcp [ 2346.676597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.764317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2346.805253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2347.856493] Key type .llcrypt unregistered [ 2347.865282] Key type ._llcrypt unregistered [ 2360.376754] Key type ._llcrypt registered [ 2360.380631] Key type .llcrypt registered [ 2361.928648] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:05:16 (1788624316) [ 2362.275454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.300888] alg: No test for adler32 (adler32-zlib) [ 2363.276667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.351359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2363.386055] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2364.220269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2364.306325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2372.411475] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:05:26 (1788624326) [ 2372.665553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.711905] LNet: 115377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.724552] LNetError: 115377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.739261] LNet: Removed LNI 192.168.204.46@tcp [ 2372.759142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.817187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2372.871721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2380.351714] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:05:34 (1788624334) [ 2380.447978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.492911] LNet: 115949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.539756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.743053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2381.293435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2381.329966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2381.379923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2381.696207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2381.725102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2381.765080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.823713] LNet: 115990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.849851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.986391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2382.440314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2382.465714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2382.512541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2382.771719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2382.805639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2391.713118] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:05:46 (1788624346) [ 2392.157375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.200654] LNet: 116591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.223532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.284696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.331135] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2393.449776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2393.475803] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2394.622273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2394.691317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2394.894935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.911814] LNetError: 116870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.935628] LNet: Removed LNI 192.168.204.46@tcp [ 2395.937653] LNetError: 116870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.962495] LNet: Removed LNI 192.168.204.46@tcp2 [ 2395.986642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.069727] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2396.075608] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2396.081603] LNet: 116876:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2396.937856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2397.350474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.393248] LNet: 116914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.399944] LNet: 116914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2398.431278] LNet: Removed LNI 192.168.204.46@tcp [ 2398.433105] LNetError: 116914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.439816] LNetError: 116914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2398.464492] LNet: Removed LNI 192.168.204.46@tcp2 [ 2398.551952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.723653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2398.777460] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2400.038299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2400.067274] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2401.508783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2401.566941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2401.729083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2401.994377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.039254] LNet: Removed LNI 192.168.204.46@tcp [ 2403.047273] LNetError: 117198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.067805] LNetError: 117198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2403.139069] LNet: Removed LNI 192.168.204.46@tcp2 [ 2403.226380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.485123] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2403.499518] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2403.511412] LNet: 117204:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2404.009226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2404.165601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2404.196756] LNet: Removed LNI 192.168.204.46@tcp [ 2405.222478] LNet: Removed LNI 192.168.204.46@tcp2 [ 2405.273743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.359042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2405.392358] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2406.312493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2406.349537] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2407.135468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2407.193867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2407.351801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2407.371654] LNet: 117521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.378918] LNet: 117521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2408.418102] LNet: Removed LNI 192.168.204.46@tcp [ 2408.435254] LNet: Removed LNI 192.168.204.46@tcp2 [ 2408.496474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.579085] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2408.585655] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2408.591612] LNet: 117527:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2409.264233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2409.443907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.463196] LNet: Removed LNI 192.168.204.46@tcp [ 2410.482616] LNet: Removed LNI 192.168.204.46@tcp2 [ 2410.520388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.591584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.634235] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2411.297924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2411.323754] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2412.121902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2412.180566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2412.276053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2412.525700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.572636] LNetError: 117849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.580359] LNetError: 117849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2412.592135] LNet: Removed LNI 192.168.204.46@tcp [ 2413.670790] LNet: Removed LNI 192.168.204.46@tcp2 [ 2413.707932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.809319] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2413.818499] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2413.824552] LNet: 117855:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.431509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2414.614883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.631306] LNet: Removed LNI 192.168.204.46@tcp [ 2415.653619] LNet: Removed LNI 192.168.204.46@tcp2 [ 2415.680632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.757460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.794358] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2416.594928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2416.618948] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2417.291412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2417.333988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2417.414367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2417.872174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.911232] LNet: Removed LNI 192.168.204.46@tcp [ 2418.929166] LNet: Removed LNI 192.168.204.46@tcp2 [ 2418.960372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.031777] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2419.040773] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2419.049439] LNet: 118196:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.497781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2419.896433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.959401] LNet: Removed LNI 192.168.204.46@tcp [ 2420.980763] LNet: Removed LNI 192.168.204.46@tcp2 [ 2421.032531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.131966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2421.167924] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2421.789415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2421.811778] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2422.572768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2422.600515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2422.630721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2422.685863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2423.054524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.095253] LNet: Removed LNI 192.168.204.46@tcp [ 2425.136052] LNet: Removed LNI 192.168.204.46@tcp2 [ 2425.192264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2425.267825] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2425.274356] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2425.951166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2434.213650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2435.933300] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:06:30 (1788624390) [ 2436.136626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.162363] LNet: 119315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.176117] LNet: 119315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2436.184558] LNetError: 119315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.191377] LNetError: 119315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2436.197304] LNet: Removed LNI 192.168.204.46@tcp [ 2437.226289] LNet: Removed LNI 192.168.204.46@tcp2 [ 2437.256774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.683838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.737342] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2438.755599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2438.779918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2439.975709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.000716] LNet: Removed LNI 10.1.2.3@tcp [ 2440.015523] LNet: Removed LNI 192.168.204.46@tcp [ 2440.050551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.125435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2440.159552] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2440.164741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2440.676995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2440.719182] LNet: Removed LNI 192.168.204.46@tcp [ 2440.742434] LNet: Removed LNI 10.1.2.3@tcp [ 2447.397298] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:06:41 (1788624401) [ 2447.537991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.629411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.724277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2447.753641] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2448.575724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2448.606993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2448.642223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.675694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.728484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.774086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.810310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.841595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.876707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.912986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.941883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2448.971618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2449.116322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2456.814099] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:06:51 (1788624411) [ 2457.085286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.152564] LNet: Removed LNI 192.168.204.46@tcp [ 2458.186828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.299903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2458.342887] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2459.065628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2459.102174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2459.167369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.217426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.272075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.331842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.382799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.442442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.504851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.545542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.583386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.626402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.793303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2459.833283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.894978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.932378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2459.982470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.016823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.058312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.115426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.149881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.185618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2460.221610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2468.210442] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:07:02 (1788624422) [ 2468.469374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2468.512480] LNet: 121686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.525306] LNet: 121686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2468.544604] LNetError: 121686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.568979] LNetError: 121686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2468.607539] LNet: Removed LNI 192.168.204.46@tcp [ 2468.675967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.837338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2469.019970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2469.031386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2469.047192] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2471.410217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2471.437606] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2472.185096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2472.207156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2473.980819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2474.045499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2474.127708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.46@tcp -m PUT -e local_timeout -r 1 [ 2474.205901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2474.314266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2474.321274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2474.571466] LNetError: 121749:0:(lib-move.c:4794:lnet_parse()) 192.168.204.46@tcp, src 192.168.204.46@tcp: Dropping PUT (error -108 looking up sender) [ 2475.616515] LNet: Removed LNI 10.1.2.3@tcp [ 2476.647226] LNet: Removed LNI 192.168.204.46@tcp [ 2477.182282] Key type .llcrypt unregistered [ 2477.184471] Key type ._llcrypt unregistered [ 2490.267722] Key type ._llcrypt registered [ 2490.270964] Key type .llcrypt registered [ 2491.782548] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:07:26 (1788624446) [ 2492.207977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2492.227914] alg: No test for adler32 (adler32-zlib) [ 2493.266076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.731742] LNet: 123058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.601666] Key type .llcrypt unregistered [ 2494.605038] Key type ._llcrypt unregistered [ 2501.265705] Key type ._llcrypt registered [ 2501.269139] Key type .llcrypt registered [ 2501.553332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2501.584249] alg: No test for adler32 (adler32-zlib) [ 2502.498333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2502.532414] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2509.628141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2509.833572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2510.643992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.146@tcp [ 2510.769590] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 2511.101538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2511.151678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2511.207085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2511.261120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2511.330849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2511.394573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2511.443808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2511.479759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2511.524753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2511.547577] LNet: 123507:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2511.570477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2511.609455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2511.654522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2511.694224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2511.755794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2511.815872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2512.682547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2512.975880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2513.003598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2513.032774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2513.066752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.100409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2513.147828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.183432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.207690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.241801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2513.262536] LNet: 123626:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.285439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2513.319847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2513.350812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2513.376248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2513.406922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2513.438636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2514.163635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.146@tcp [ 2514.417253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2514.455918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2514.542837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2514.615808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2514.682939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2514.739887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2514.791383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2514.826974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2514.876852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2514.903476] LNet: 123745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2514.942256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2514.984336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2515.027403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2515.068543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2515.107469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2515.172791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2515.824718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2516.137306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2516.168556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2516.200612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2516.236392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2516.267429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2516.294839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2516.323904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2516.350880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2516.387464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2516.412926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2516.440960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2516.476796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2516.504371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2516.533744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2516.576538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2516.995805] LNet: 123911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.005375] LNetError: 123911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2517.017911] LNet: Removed LNI 192.168.204.46@tcp [ 2517.428951] Key type .llcrypt unregistered [ 2517.435440] Key type ._llcrypt unregistered [ 2534.368748] Key type ._llcrypt registered [ 2534.371602] Key type .llcrypt registered [ 2535.966863] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:08:10 (1788624490) [ 2536.548428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.562905] alg: No test for adler32 (adler32-zlib) [ 2537.462151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2537.784788] LNet: 124795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.339685] Key type .llcrypt unregistered [ 2538.341440] Key type ._llcrypt unregistered [ 2545.609929] Key type ._llcrypt registered [ 2545.612991] Key type .llcrypt registered [ 2545.924473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2545.942551] alg: No test for adler32 (adler32-zlib) [ 2546.828242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2546.856477] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2555.671424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2566.296380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.354552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2566.385203] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 2567.237615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2567.994178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.146@tcp [ 2568.247096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2568.321431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2568.351182] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 2568.394560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2568.444251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2568.491844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2568.542174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2568.584498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2568.634824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2568.670960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2568.703863] LNet: 125446:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2568.729623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2568.785438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2568.837712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2568.908276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2568.958825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2568.994863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2569.721766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2569.963890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2569.989677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2570.016316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2570.048163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2570.078375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2570.103798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2570.134742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2570.183877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2570.238539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2570.253531] LNet: 125566:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2570.268362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2570.299097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2570.327956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2570.355317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2570.390615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2570.454674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2571.239445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.146@tcp [ 2571.481963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2571.513351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2571.573952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2571.616414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2571.662771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2571.712108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2571.786347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2571.821872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2571.854791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2571.872426] LNet: 125685:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2571.902307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2571.952205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2572.024703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2572.081667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2572.125730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2572.155741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2572.817826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2573.133372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2573.164455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2573.209636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2573.251783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2573.292814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2573.325200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2573.359567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2573.413184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2573.440812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2573.476950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2573.515868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2573.556255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2573.601181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2573.641769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2573.686287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2578.817691] LNet: 125885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.837558] LNetError: 125885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.857465] LNet: Removed LNI 192.168.204.46@tcp [ 2578.869719] LNet: Removed LNI 192.168.204.46@tcp1 [ 2579.352856] Key type .llcrypt unregistered [ 2579.361649] Key type ._llcrypt unregistered [ 2596.603631] Key type ._llcrypt registered [ 2596.606225] Key type .llcrypt registered [ 2598.304233] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:09:12 (1788624552) [ 2598.862830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2598.897809] alg: No test for adler32 (adler32-zlib) [ 2599.971454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2600.044273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2600.071204] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2600.992941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2601.018964] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2602.198498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2602.272622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2603.359193] LNet: Removed LNI 192.168.204.46@tcp [ 2611.213739] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:09:25 (1788624565) [ 2613.409399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2615.721601] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:09:30 (1788624570) [ 2625.361456] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:09:39 (1788624579) [ 2635.720964] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:09:50 (1788624590) [ 2644.741968] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:09:58 (1788624598) [ 2653.302535] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:10:07 (1788624607) [ 2659.985666] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:10:14 (1788624614) [ 2669.486980] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:10:23 (1788624623) [ 2678.920649] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:10:33 (1788624633) [ 2687.917892] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:10:42 (1788624642) [ 2695.340765] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:10:49 (1788624649) [ 2703.200045] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:10:57 (1788624657) [ 2711.485124] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:11:05 (1788624665) [ 2718.683757] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:11:13 (1788624673) [ 2727.978471] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:11:22 (1788624682) [ 2737.728876] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:11:31 (1788624691) [ 2747.439670] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:11:41 (1788624701) [ 2755.531312] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:11:50 (1788624710) [ 2764.032977] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:11:58 (1788624718) [ 2764.194403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2764.237327] LNet: 137575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.250447] LNetError: 137575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2764.292310] LNet: Removed LNI 192.168.204.46@tcp2 [ 2764.369765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2764.578504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2764.619465] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2766.132194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2766.151622] LNet: 137723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.156239] LNetError: 137723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.165808] LNet: Removed LNI 192.168.204.46@tcp [ 2766.183225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2766.255402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2766.274337] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2775.078742] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:12:08 (1788624728) [ 2782.884405] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:12:17 (1788624737) [ 2791.150677] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:12:25 (1788624745) [ 2798.165796] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:12:32 (1788624752) [ 2804.469660] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:12:39 (1788624759) [ 2811.327029] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:12:45 (1788624765) [ 2819.374568] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:12:53 (1788624773) [ 2825.970992] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:13:00 (1788624780) [ 2834.198557] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:13:08 (1788624788) [ 2842.491703] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:13:16 (1788624796) [ 2850.682045] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:13:25 (1788624805) [ 2857.480419] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:13:32 (1788624812) [ 2863.027342] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2621 sec ========== 12:13:37 (1788624817) [ 2863.293893] LNet: 145188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2863.298734] LNetError: 145188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2863.318962] LNet: Removed LNI 192.168.204.46@tcp [ 2863.973856] Key type .llcrypt unregistered [ 2863.978248] Key type ._llcrypt unregistered [ 2874.801243] Key type ._llcrypt registered [ 2874.826330] Key type .llcrypt registered [ 2875.198673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2875.218492] alg: No test for adler32 (adler32-zlib) [ 2876.312680] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2876.598381] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2878.287476] Key type lgssc registered [ 2879.789611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2944.822791] Lustre: Mounted lustre-client - version 2.17.58_2_gf69a9da [ 2950.286592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2957.237290] Lustre: Unmounted lustre-client [ 2976.297025] Key type lgssc unregistered [ 2976.543254] LNet: 148134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.558485] LNetError: 148134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2976.575528] LNet: Removed LNI 192.168.204.46@tcp [ 2977.287321] Key type .llcrypt unregistered [ 2977.289097] Key type ._llcrypt unregistered