[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489272100 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002408] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007521] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.020363] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025280] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] *** VALIDATE tmpfs *** [ 0.029000] *** VALIDATE proc *** [ 0.029248] *** VALIDATE cgroup *** [ 0.030014] *** VALIDATE cgroup2 *** [ 0.032055] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.039218] debug: unmapping init [mem 0xffffffff9d059000-0xffffffff9d060fff] [ 0.042174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043854] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044034] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046026] ... generic registers: 4 [ 0.047021] ... value mask: 0000ffffffffffff [ 0.048019] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050019] ... event mask: 000000070000000f [ 0.051380] rcu: Hierarchical SRCU implementation. [ 0.053450] smp: Bringing up secondary CPUs ... [ 0.054563] x86: Booting SMP configuration: [ 0.055018] .... node #0, CPUs: #1 #2 #3 [ 0.059108] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117893] node 0 deferred pages initialised in 53ms [ 0.120110] devtmpfs: initialized [ 0.122299] x86/mm: Memory block size: 128MB [ 0.124000] gcov: version magic: 0x41383552 [ 0.126331] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.127081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128259] pinctrl core: initialized pinctrl subsystem [ 0.129211] [ 0.129764] ************************************************************* [ 0.130016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131013] ** ** [ 0.132021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133020] ** ** [ 0.134016] ** This means that this kernel is built to expose internal ** [ 0.135013] ** IOMMU data structures, which may compromise security on ** [ 0.136015] ** your system. ** [ 0.137013] ** ** [ 0.138016] ** If you see this message and you are not debugging the ** [ 0.139012] ** kernel, report this immediately to your vendor! ** [ 0.140015] ** ** [ 0.141013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142015] ************************************************************* [ 0.144171] NET: Registered protocol family 16 [ 0.145637] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.146068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.147060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.149321] cpuidle: using governor menu [ 0.151000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.155604] PCI: Using configuration type 1 for base access [ 0.158248] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178141] cryptd: max_cpu_qlen set to 1000 [ 0.182228] ACPI: Added _OSI(Module Device) [ 0.183016] ACPI: Added _OSI(Processor Device) [ 0.185012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192799] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199134] ACPI: Interpreter enabled [ 0.201093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203018] ACPI: Using IOAPIC for interrupt routing [ 0.205112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222614] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226139] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229037] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234167] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240841] acpiphp: Slot [2] registered [ 0.242365] acpiphp: Slot [5] registered [ 0.244352] acpiphp: Slot [6] registered [ 0.246118] acpiphp: Slot [3] registered [ 0.248170] acpiphp: Slot [4] registered [ 0.250099] acpiphp: Slot [7] registered [ 0.252107] acpiphp: Slot [8] registered [ 0.253109] acpiphp: Slot [9] registered [ 0.255095] acpiphp: Slot [10] registered [ 0.257098] acpiphp: Slot [11] registered [ 0.258098] acpiphp: Slot [12] registered [ 0.260105] acpiphp: Slot [13] registered [ 0.262094] acpiphp: Slot [14] registered [ 0.264099] acpiphp: Slot [15] registered [ 0.266071] acpiphp: Slot [16] registered [ 0.267182] acpiphp: Slot [17] registered [ 0.269104] acpiphp: Slot [18] registered [ 0.271083] acpiphp: Slot [19] registered [ 0.272127] acpiphp: Slot [20] registered [ 0.274116] acpiphp: Slot [21] registered [ 0.276085] acpiphp: Slot [22] registered [ 0.278087] acpiphp: Slot [23] registered [ 0.280117] acpiphp: Slot [24] registered [ 0.281100] acpiphp: Slot [25] registered [ 0.283081] acpiphp: Slot [26] registered [ 0.285099] acpiphp: Slot [27] registered [ 0.287094] acpiphp: Slot [28] registered [ 0.288206] acpiphp: Slot [29] registered [ 0.290238] acpiphp: Slot [30] registered [ 0.292162] acpiphp: Slot [31] registered [ 0.294059] PCI host bridge to bus 0000:00 [ 0.295016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300111] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307067] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317260] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320532] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327024] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.333077] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338048] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340054] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342055] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346531] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349977] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356922] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.373023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385511] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439382] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506176] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.512372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.518422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.524470] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.528000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.536112] iommu: Default domain type: Passthrough [ 0.537454] SCSI subsystem initialized [ 0.538144] ACPI: bus type USB registered [ 0.542173] usbcore: registered new interface driver usbfs [ 0.546108] usbcore: registered new interface driver hub [ 0.550100] usbcore: registered new device driver usb [ 0.553547] pps_core: LinuxPPS API ver. 1 registered [ 0.557010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.563071] PTP clock support registered [ 0.567325] EDAC MC: Ver: 3.0.0 [ 0.571282] PCI: Using ACPI for IRQ routing [ 0.576105] NetLabel: Initializing [ 0.579013] NetLabel: domain hash size = 128 [ 0.582016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.586084] NetLabel: unlabeled traffic allowed by default [ 0.589763] vgaarb: loaded [ 0.592289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.605379] clocksource: Switched to clocksource kvm-clock [ 0.758437] VFS: Disk quotas dquot_6.6.0 [ 0.759826] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.762162] *** VALIDATE ramfs *** [ 0.763568] *** VALIDATE hugetlbfs *** [ 0.765151] pnp: PnP ACPI init [ 0.767746] pnp: PnP ACPI: found 6 devices [ 0.788156] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.791640] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.793299] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795070] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.797743] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.800398] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.803247] NET: Registered protocol family 2 [ 0.805783] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.811035] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.814922] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820361] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.824114] TCP: Hash tables configured (established 65536 bind 65536) [ 0.827307] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.830617] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.833787] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837366] NET: Registered protocol family 1 [ 0.841292] RPC: Registered named UNIX socket transport module. [ 0.846141] RPC: Registered udp transport module. [ 0.849481] RPC: Registered tcp transport module. [ 0.853712] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.857248] NET: Registered protocol family 44 [ 0.861805] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865595] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.871708] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.876432] PCI: CLS 0 bytes, default 64 [ 0.880807] Unpacking initramfs... [ 3.059327] debug: unmapping init [mem 0xffff917d7cc64000-0xffff917d7ffcffff] [ 3.065398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.068760] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.072595] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.684909] Initialise system trusted keyrings [ 3.687113] Key type blacklist registered [ 3.691825] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.702808] zbud: loaded [ 3.708514] *** VALIDATE nfs *** [ 3.710089] *** VALIDATE nfs4 *** [ 3.716528] pstore: using deflate compression [ 3.726763] Platform Keyring initialized [ 3.865160] NET: Registered protocol family 38 [ 3.870559] Key type asymmetric registered [ 3.874460] Asymmetric key parser 'x509' registered [ 3.878648] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.885361] io scheduler mq-deadline registered [ 3.886978] io scheduler kyber registered [ 3.889038] io scheduler bfq registered [ 3.891858] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.894549] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.897153] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.899668] ACPI: Power Button [PWRF] [ 3.904312] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.912363] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.926284] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.952314] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.981558] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.987678] Non-volatile memory driver v1.3 [ 3.989798] Linux agpgart interface v0.103 [ 4.024253] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.027712] vda: detected capacity change from 0 to 76787712 [ 4.053754] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.058489] vdb: detected capacity change from 0 to 1073741824 [ 4.075863] libphy: Fixed MDIO Bus: probed [ 4.097804] usbcore: registered new interface driver usbserial_generic [ 4.100982] usbserial: USB Serial support registered for generic [ 4.104301] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.110397] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.113159] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.117301] mousedev: PS/2 mouse device common for all mice [ 4.122377] rtc_cmos 00:05: RTC can wake from S4 [ 4.125572] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.130279] rtc_cmos 00:05: registered as rtc0 [ 4.133398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.134295] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.134334] intel_pstate: CPU model not supported [ 4.137111] hid: raw HID events driver (C) Jiri Kosina [ 4.141919] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.146231] usbcore: registered new interface driver usbhid [ 4.146238] usbhid: USB HID core driver [ 4.146373] drop_monitor: Initializing network drop monitor service [ 4.146504] Initializing XFRM netlink socket [ 4.148107] NET: Registered protocol family 10 [ 4.167309] Segment Routing with IPv6 [ 4.168938] NET: Registered protocol family 17 [ 4.171097] mpls_gso: MPLS GSO support [ 4.176926] RAS: Correctable Errors collector initialized. [ 4.179248] AVX version of gcm_enc/dec engaged. [ 4.180950] AES CTR mode by8 optimization enabled [ 4.290563] sched_clock: Marking stable (4290421946, 0)->(5256034399, -965612453) [ 4.295320] registered taskstats version 1 [ 4.297944] Loading compiled-in X.509 certificates [ 4.300863] zswap: loaded using pool lzo/zbud [ 4.336789] Key type big_key registered [ 4.353654] Key type encrypted registered [ 4.355906] ima: No TPM chip found, activating TPM-bypass! [ 4.360198] ima: Allocated hash algorithm: sha1 [ 4.362899] ima: No architecture policies found [ 4.368309] evm: Initialising EVM extended attributes: [ 4.372156] evm: security.selinux [ 4.375299] evm: security.ima [ 4.377679] evm: security.capability [ 4.381362] evm: HMAC attrs: 0x1 [ 4.386903] rtc_cmos 00:05: setting system clock to 2026-09-17 22:11:10 UTC (1789683070) [ 4.397235] debug: unmapping init [mem 0xffffffff9e003000-0xffffffff9e1fffff] [ 4.402930] debug: unmapping init [mem 0xffffffff9cd82000-0xffffffff9d058fff] [ 4.415371] Write protecting the kernel read-only data: 28672k [ 4.424049] debug: unmapping init [mem 0xffffffff9b403000-0xffffffff9b5fffff] [ 4.429235] debug: unmapping init [mem 0xffffffff9bd14000-0xffffffff9bdfffff] [ 4.480513] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.495820] systemd[1]: Detected virtualization kvm. [ 4.499369] systemd[1]: Detected architecture x86-64. [ 4.502908] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.545423] systemd[1]: No hostname configured. [ 4.549725] systemd[1]: Set hostname to . [ 4.551237] random: systemd: uninitialized urandom read (16 bytes read) [ 4.556356] systemd[1]: Initializing machine ID from random generator. [ 4.750386] random: ln: uninitialized urandom read (6 bytes read) [ 5.076830] random: systemd: uninitialized urandom read (16 bytes read) [ 5.081223] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.098533] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.107898] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.411640] device-mapper: uevent: version 1.0.3 [ 6.415537] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.820661] virtio_net virtio0 ens2: renamed from eth0 [ 8.152816] scsi host0: ata_piix [ 8.479994] scsi host1: ata_piix [ 8.481974] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.489405] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.899830] random: crng init done [ 13.906131] random: 7 urandom warning(s) missed due to ratelimiting [ 16.749697] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [ 18.065368] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.120694] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.644949] SELinux: Disabled at runtime. [ 20.752657] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.769564] systemd[1]: Detected virtualization kvm. [ 20.772170] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.244236] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.256802] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.287293] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.302184] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.315858] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.338074] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.349921] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ 22.843663] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.416967] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.607903] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.620568] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.310666] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.440178] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 30.813178] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.599958] NFS: Registering the id_resolver key type [ 31.603356] Key type id_resolver registered [ 31.605470] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-client login: [ 96.704253] libcfs: loading out-of-tree module taints kernel. [ 96.827108] Key type ._llcrypt registered [ 96.829595] Key type .llcrypt registered [ 97.330263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 97.344800] alg: No test for adler32 (adler32-zlib) [ 98.644178] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 99.319160] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 101.055647] Key type lgssc registered [ 102.552527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.619644] hrtimer: interrupt took 4392721 ns [ 238.831103] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a [ 244.598344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.427913] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing check_logdir /tmp/testlogs/ [ 264.672064] Lustre: lustre-OST0000-osc-ffff917dd0d92800: disconnect after 23s idle [ 265.266334] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing yml_node [ 269.970428] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 272.900259] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 275.536319] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 277.168049] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 17 18:15:41 EDT 2026 [ 293.826475] Lustre: Unmounted lustre-client [ 322.870951] Key type lgssc unregistered [ 323.278953] LNet: 8427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.297233] LNetError: 8427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.332961] LNet: Removed LNI 192.168.204.11@tcp [ 324.089174] Key type .llcrypt unregistered [ 324.093087] Key type ._llcrypt unregistered [ 354.284684] Key type ._llcrypt registered [ 354.287570] Key type .llcrypt registered [ 354.579595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 354.602670] alg: No test for adler32 (adler32-zlib) [ 355.523164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 355.585719] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 355.605826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 360.406075] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 360.627317] LNet: 9308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.639186] LNetError: 9308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 360.652800] LNet: Removed LNI 192.168.204.11@tcp [ 361.430622] Key type .llcrypt unregistered [ 361.435244] Key type ._llcrypt unregistered [ 367.793412] Key type ._llcrypt registered [ 367.795310] Key type .llcrypt registered [ 369.287076] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:17:13 (1789683433) [ 369.687966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 369.714139] alg: No test for adler32 (adler32-zlib) [ 370.607446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.747293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.782213] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.809238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.844847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 370.859466] LNet: 9723:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 377.572169] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:17:22 (1789683442) [ 377.712446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.742954] LNet: 10303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.769885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.905482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 384.058274] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:17:29 (1789683449) [ 384.505521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.522804] LNet: 10888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.544928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.628909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 391.397918] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:17:35 (1789683455) [ 391.521749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.553298] LNet: 11468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.589404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.692987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 398.214788] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:17:42 (1789683462) [ 398.339052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.368450] LNet: 12053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.400590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.500786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 399.191265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 406.479597] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:17:51 (1789683471) [ 406.788541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.830870] LNet: 12648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.850244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.001811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 407.454325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 408.041574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 415.616865] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:18:00 (1789683480) [ 415.676178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.725096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.821569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 421.924487] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:18:06 (1789683486) [ 421.991484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.036874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.164699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 422.235682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 422.721780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 422.761879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 423.195974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 423.252058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 423.645515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 423.696389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 424.022091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 424.044215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 424.711812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 424.777899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.476356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 425.520682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.838619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 425.863741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.153976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 426.184660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.668852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 426.710588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 427.603580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 427.632885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.115933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 428.205886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 428.626192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 428.663255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 436.154130] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:18:20 (1789683500) [ 436.298931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.354697] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.364392] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 436.396979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.557849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 436.646825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 445.999562] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:18:30 (1789683510) [ 446.303317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.406603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.575391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 446.627350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 454.954536] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:18:39 (1789683519) [ 455.072477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.145219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.263388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 455.295430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 462.960125] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:18:47 (1789683527) [ 463.025698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.073286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.166344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 463.200798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.921908] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:18:55 (1789683535) [ 471.199857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.258068] LNet: 16857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.263331] LNet: 16857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 471.330853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.426090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 471.462091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 478.713066] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:19:03 (1789683543) [ 479.095565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.175692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.266984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 479.323529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.784760] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:19:11 (1789683551) [ 488.069554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.123467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.255776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.831260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.924801] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:19:21 (1789683561) [ 497.049063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.117697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.462410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.857326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.367693] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:19:30 (1789683570) [ 506.549666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.646600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.802478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.088346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.707913] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:19:39 (1789683579) [ 514.969377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.013918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.436769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.779337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.927265] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:19:48 (1789683588) [ 524.097281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.191656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.362592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.441977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.337861] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:19:57 (1789683597) [ 532.605505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.667349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.763238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.818606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.322788] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:20:04 (1789683604) [ 539.445606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.477509] LNet: 21713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.484760] LNet: 21713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 539.531537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.619167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 546.354043] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:20:11 (1789683611) [ 546.585219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.625431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.723183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 553.931197] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:20:18 (1789683618) [ 554.028617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.101827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.263419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 554.772709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 561.254487] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:20:25 (1789683625) [ 561.585670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.628689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.717621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 561.763549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 568.277975] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:20:33 (1789683633) [ 568.418413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.464486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.569526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 568.613743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 575.505570] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:20:39 (1789683639) [ 575.789682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.840690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.920360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 575.963266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 583.566753] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:20:48 (1789683648) [ 583.664300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.719821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.839635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.878461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 591.243803] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:20:55 (1789683655) [ 591.402400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.468923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.572765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 591.654629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 591.707540] LNetError: 25831:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 591.792614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 591.841878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 598.265264] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:21:03 (1789683663) [ 598.450768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.505224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.605399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xKw9X8OpX5 [ 598.714845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xKw9X8OpX5 [ 598.812201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xKw9X8OpX5 [ 605.794945] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:21:10 (1789683670) [ 605.868390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.915411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.996691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 606.026840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 606.066651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 613.277576] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 18:21:17 (1789683677) [ 613.361758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.398818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.460649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.485252] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 614.865378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 616.193081] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 618.215297] Key type lgssc registered [ 619.755843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.534508] LNetError: 27588:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.11@tcp99 has no available nets [ 626.287405] Key type lgssc unregistered [ 626.659994] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.678933] LNet: Removed LNI 192.168.204.11@tcp [ 627.327507] Key type .llcrypt unregistered [ 627.334159] Key type ._llcrypt unregistered [ 633.723078] Key type ._llcrypt registered [ 633.725676] Key type .llcrypt registered [ 635.152654] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:21:39 (1789683699) [ 635.827305] Key type .llcrypt unregistered [ 635.831488] Key type ._llcrypt unregistered [ 642.072110] Key type ._llcrypt registered [ 642.074985] Key type .llcrypt registered [ 642.215866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.245897] alg: No test for adler32 (adler32-zlib) [ 643.124230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.189226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 643.209257] LNet: 29609:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 643.213763] LNet: 29609:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 643.218237] LNet: 29609:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 643.232874] LNet: Message forwarding will be enabled in 2s [ 646.303263] LNet: Message forwarding enabled [ 646.643223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.673374] LNet: Message forwarding will stop in 2s to 6s [ 650.401559] LNet: Message forwarding disabled [ 651.277288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 651.330777] LNet: 29707:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 651.355976] LNet: 29707:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 651.362650] LNet: 29707:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 651.388885] LNet: Message forwarding will be enabled in 2s [ 651.614839] LNet: 29720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.391680] Key type .llcrypt unregistered [ 652.394468] Key type ._llcrypt unregistered [ 666.806808] Key type ._llcrypt registered [ 666.809954] Key type .llcrypt registered [ 668.533635] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:22:12 (1789683732) [ 669.175061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.203861] alg: No test for adler32 (adler32-zlib) [ 670.093821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.166504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 676.336661] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:22:21 (1789683741) [ 676.486810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.530708] LNet: 31112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.579363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.722484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 676.761934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 676.797194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 676.846634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 676.888886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 676.922337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 676.974171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 677.033943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 677.091244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 677.120357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 677.149517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 677.185244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 677.213307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 677.245437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 677.286057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 677.335711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 677.373937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 677.420103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 677.457777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 677.490724] LNetError: 31163:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 684.579870] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:22:29 (1789683749) [ 684.705193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.727513] LNet: 31730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.765404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.878630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.860047] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:22:36 (1789683756) [ 693.992592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg411-server [ 696.131391] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:22:40 (1789683760) [ 696.471552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.515875] LNet: 32667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.549190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.618180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.641991] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 697.963611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[12-16]@tcp [ 698.016711] LNet: 32800:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 698.134103] LNetError: 32671:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 698.159072] LNetError: 32671:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.15@tcp has gone from up to down [ 705.993511] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:22:50 (1789683770) [ 707.942680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg411-server [ 709.800968] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:22:54 (1789683774) [ 709.897040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.930262] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.940767] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.961668] LNet: Removed LNI 192.168.204.11@tcp [ 710.002824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.123140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.177677] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 710.989403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 711.015186] LNet: 33855:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 711.021764] LNet: 33855:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 711.039667] LNetError: 33742:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 711.051561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 711.055621] LNetError: 33742:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 711.055660] LNetError: 33742:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.16@tcp has gone from up to down [ 711.055665] LNetError: 33742:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 717.902576] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:23:02 (1789683782) [ 718.176385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.243490] LNet: 34447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.253688] LNetError: 34447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.288450] LNet: Removed LNI 192.168.204.11@tcp [ 718.314139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.419885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 718.509123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.573544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.626387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 718.686861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.720159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.775657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 718.852365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.896604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.961349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.051295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.095446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.152788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 719.227641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.275815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.358578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.499872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.557913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.613726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 719.687594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.726888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.814210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.899783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.934556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.015253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 720.093853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.125294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.208164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 720.306815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.353966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.407828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 720.468541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.506601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.582840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.635629] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:23:12 (1789683792) [ 729.905366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg411-server [ 732.710742] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:23:16 (1789683796) [ 734.658386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg411-server [ 736.434974] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:23:21 (1789683801) [ 736.709478] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.718963] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 737.480882] Key type .llcrypt unregistered [ 737.482718] Key type ._llcrypt unregistered [ 743.565784] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 743.724764] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 743.733808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 743.739249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 746.337235] Key type ._llcrypt registered [ 746.340870] Key type .llcrypt registered [ 746.558529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.576386] alg: No test for adler32 (adler32-zlib) [ 747.526029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.598924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.638967] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 748.435642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 748.467689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.652558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 750.668307] LNet: Removed LNI 10.1.2.3@tcp [ 750.758838] LNet: 36565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.766349] LNetError: 36565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.780274] LNet: Removed LNI 192.168.204.11@tcp [ 751.324499] Key type .llcrypt unregistered [ 751.328122] Key type ._llcrypt unregistered [ 761.290781] Key type ._llcrypt registered [ 761.294189] Key type .llcrypt registered [ 762.694924] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:23:47 (1789683827) [ 763.288340] Key type .llcrypt unregistered [ 763.290439] Key type ._llcrypt unregistered [ 768.459883] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.639296] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.647187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.651891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.206412] Key type ._llcrypt registered [ 771.209581] Key type .llcrypt registered [ 771.356357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.368474] alg: No test for adler32 (adler32-zlib) [ 772.283590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.333901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.361599] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 773.090519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 773.114240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.654788] LNet: 38052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.661289] LNetError: 38052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 775.667804] LNet: Removed LNI 10.1.2.3@tcp [ 775.682671] LNet: Removed LNI 192.168.204.11@tcp [ 776.307141] Key type .llcrypt unregistered [ 776.309071] Key type ._llcrypt unregistered [ 786.248374] Key type ._llcrypt registered [ 786.252154] Key type .llcrypt registered [ 787.723095] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:24:12 (1789683852) [ 788.312400] Key type .llcrypt unregistered [ 788.315487] Key type ._llcrypt unregistered [ 794.208747] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 794.527886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 794.532519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 794.538630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 797.176867] Key type ._llcrypt registered [ 797.180603] Key type .llcrypt registered [ 797.507650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 797.552317] alg: No test for adler32 (adler32-zlib) [ 798.524287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.598991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 798.633474] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 800.672542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.703272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.275712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 803.296936] LNet: Removed LNI 10.1.2.3@tcp [ 803.322061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 803.368566] LNet: Removed LNI 10.1.2.31@tcp [ 803.387258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 803.408128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.216306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 804.240954] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 805.467789] LNet: 39968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.475590] LNetError: 39968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 805.483837] LNet: Removed LNI 10.1.2.31@tcp [ 805.498430] LNet: Removed LNI 10.1.2.3@tcp [ 806.201044] Key type .llcrypt unregistered [ 806.205510] Key type ._llcrypt unregistered [ 819.178595] Key type ._llcrypt registered [ 819.181421] Key type .llcrypt registered [ 821.016449] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:24:45 (1789683885) [ 821.579159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 821.601232] alg: No test for adler32 (adler32-zlib) [ 822.659877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.730146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.772549] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 823.725365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.760802] LNet: 40917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.768654] LNetError: 40917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.807564] LNet: Removed LNI 192.168.204.11@tcp [ 823.841104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.918519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.11@tcp [ 823.971843] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 824.988371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.11@tcp [ 825.025549] LNet: Removed LNI 192.168.204.11@tcp [ 825.113796] LNet: 41053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.949413] Key type .llcrypt unregistered [ 825.955741] Key type ._llcrypt unregistered [ 839.691527] Key type ._llcrypt registered [ 839.694308] Key type .llcrypt registered [ 841.322869] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:25:05 (1789683905) [ 843.522342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg411-server [ 845.431272] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:25:10 (1789683910) [ 846.098756] Key type .llcrypt unregistered [ 846.109150] Key type ._llcrypt unregistered [ 853.009224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 853.282439] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 853.286563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.301452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.997495] Key type ._llcrypt registered [ 856.000718] Key type .llcrypt registered [ 856.201994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.215137] alg: No test for adler32 (adler32-zlib) [ 857.084358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.160629] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 857.990879] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 860.075888] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.082479] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.088148] LNet: Removed LNI 192.168.204.11@tcp1 [ 860.097305] LNet: Removed LNI 10.1.2.3@tcp2 [ 860.617321] Key type .llcrypt unregistered [ 860.620765] Key type ._llcrypt unregistered [ 872.128497] Key type ._llcrypt registered [ 872.130601] Key type .llcrypt registered [ 873.415161] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:25:38 (1789683938) [ 873.950987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.984505] alg: No test for adler32 (adler32-zlib) [ 874.939748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.018964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.036388] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 875.924773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.942571] LNet: 43849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.951078] LNetError: 43849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.971476] LNet: Removed LNI 192.168.204.11@tcp [ 876.013485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.086381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 876.131737] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 883.194663] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:25:48 (1789683948) [ 883.631458] LNet: 44440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.641257] LNetError: 44440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.660259] LNet: Removed LNI 192.168.204.11@tcp [ 884.350491] Key type .llcrypt unregistered [ 884.353814] Key type ._llcrypt unregistered [ 891.011910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.251829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.261539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.274911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.882616] Key type ._llcrypt registered [ 893.884183] Key type .llcrypt registered [ 894.039734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.060583] alg: No test for adler32 (adler32-zlib) [ 895.024234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.114733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.140266] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 895.831845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 895.857818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.077313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.114967] LNet: 45180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.123300] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.129951] LNet: Removed LNI 10.1.2.3@tcp [ 898.146421] LNet: Removed LNI 192.168.204.11@tcp [ 898.178923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.253049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.283551] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 898.291373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.758801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.773494] LNet: 45207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.785990] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.796951] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.803806] LNet: Removed LNI 10.1.2.3@tcp [ 898.812925] LNet: Removed LNI 192.168.204.11@tcp [ 898.871437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.963934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.004396] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 899.009824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.588760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 899.608658] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 900.737791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.770546] LNet: 45369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.785787] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.794075] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 900.801796] LNet: Removed LNI 10.1.2.3@tcp [ 900.805231] LNet: Removed LNI 192.168.204.11@tcp [ 900.825565] LNet: Removed LNI 192.168.204.11@tcp2 [ 900.847562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.915539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 900.940534] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 900.945768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.950987] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 908.451807] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:26:13 (1789683973) [ 908.837382] LNet: 45968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.846117] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.852972] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 908.866405] LNet: Removed LNI 10.1.2.3@tcp [ 908.874256] LNet: Removed LNI 192.168.204.11@tcp [ 908.892231] LNet: Removed LNI 192.168.204.11@tcp2 [ 909.433910] Key type .llcrypt unregistered [ 909.437427] Key type ._llcrypt unregistered [ 915.227689] Key type ._llcrypt registered [ 915.229909] Key type .llcrypt registered [ 915.429962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.452590] alg: No test for adler32 (adler32-zlib) [ 916.352978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.401871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.421381] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 917.266974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.317768] LNet: 46359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.324078] LNetError: 46359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.355182] LNet: Removed LNI 192.168.204.11@tcp [ 917.398432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.502846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.552155] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 918.190438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.211120] LNet: 46389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.216449] LNetError: 46389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.237644] LNet: Removed LNI 192.168.204.11@tcp [ 918.272489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.362923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 918.394572] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 919.010820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.072725] LNet: Removed LNI 192.168.204.11@tcp [ 919.107714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.198425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 919.224298] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 919.778903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.930250] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.935359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.938798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.277649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.297613] LNet: 46492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.302592] LNet: 46492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 922.311049] LNetError: 46492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.321659] LNetError: 46492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.345251] LNet: Removed LNI 192.168.204.11@tcp [ 922.384091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.458760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 922.495553] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 928.265105] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:26:33 (1789683993) [ 928.369202] LNet: 47081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.380309] LNetError: 47081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.402094] LNet: Removed LNI 192.168.204.11@tcp [ 929.051456] Key type .llcrypt unregistered [ 929.056034] Key type ._llcrypt unregistered [ 935.042336] Key type ._llcrypt registered [ 935.046982] Key type .llcrypt registered [ 935.283567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.296589] alg: No test for adler32 (adler32-zlib) [ 936.208692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.276646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.302410] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 937.290346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.312252] LNet: 47484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.317938] LNetError: 47484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.332336] LNet: Removed LNI 192.168.204.11@tcp [ 937.360370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.450105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 937.481977] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 943.869454] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:26:48 (1789684008) [ 944.193871] LNet: 48076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.201476] LNetError: 48076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.220067] LNet: Removed LNI 192.168.204.11@tcp [ 944.842227] Key type .llcrypt unregistered [ 944.849473] Key type ._llcrypt unregistered [ 950.345526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 950.563835] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 950.567348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 950.571491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 953.066855] Key type ._llcrypt registered [ 953.070236] Key type .llcrypt registered [ 953.249317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.275799] alg: No test for adler32 (adler32-zlib) [ 954.204326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.277073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.312834] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 955.267097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.328327] LNet: 48521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.337721] LNetError: 48521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.370367] LNet: Removed LNI 192.168.204.11@tcp [ 955.411090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.498978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 955.560621] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 962.426918] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:27:07 (1789684027) [ 962.546901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.580522] LNet: 49113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.594622] LNetError: 49113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.614456] LNet: Removed LNI 192.168.204.11@tcp [ 962.644971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.786971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 969.079678] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:27:13 (1789684033) [ 969.268584] LNet: 49703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.818239] Key type .llcrypt unregistered [ 969.820996] Key type ._llcrypt unregistered [ 974.719716] Key type ._llcrypt registered [ 974.722245] Key type .llcrypt registered [ 974.894636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.909863] alg: No test for adler32 (adler32-zlib) [ 975.823673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.868443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.886472] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 976.644827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.677294] LNet: 50097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.690830] LNetError: 50097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.712611] LNet: Removed LNI 192.168.204.11@tcp [ 976.740280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.822586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 976.871597] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 981.893784] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:27:27 (1789684047) [ 982.050350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.076904] LNet: 50688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.081343] LNetError: 50688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.114870] LNet: Removed LNI 192.168.204.11@tcp [ 982.141249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.199362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 986.762378] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:27:31 (1789684051) [ 988.039390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 989.847276] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:27:34 (1789684054) [ 991.271317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 992.742851] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:27:37 (1789684057) [ 994.091269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 995.850944] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:27:40 (1789684060) [ 997.375989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 999.006729] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:27:43 (1789684063) [ 1000.737647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1002.507028] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:27:47 (1789684067) [ 1002.599182] LNet: 52702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.264525] Key type .llcrypt unregistered [ 1003.268912] Key type ._llcrypt unregistered [ 1009.838573] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.007412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.010168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.013750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1012.533700] Key type ._llcrypt registered [ 1012.537086] Key type .llcrypt registered [ 1012.802874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.850814] alg: No test for adler32 (adler32-zlib) [ 1013.750634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.850564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1013.871740] LNet: Added LNI 192.168.204.11@tcp [8/512/0/180] [ 1013.884803] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1013.887694] LNet: 53030:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.003866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.024654] LNet: 53049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.028778] LNetError: 53049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.033348] LNet: Removed LNI 10.1.2.3@tcp [ 1014.034980] LNetError: 53049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.047521] LNet: Removed LNI 192.168.204.11@tcp [ 1014.063435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.145802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.178868] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1014.270529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.293661] LNet: 53073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.301646] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.319868] LNet: Removed LNI 192.168.204.11@tcp [ 1014.345908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.413435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.432527] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1014.436876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1014.442247] LNet: 53081:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.526689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.547385] LNet: 53099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.552801] LNetError: 53099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.558354] LNet: Removed LNI 10.1.2.3@tcp [ 1014.560651] LNetError: 53099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.571677] LNet: Removed LNI 192.168.204.11@tcp [ 1014.594102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.655409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.685165] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1014.689652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1014.692996] LNet: 53107:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1014.920883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.939951] LNet: 53133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.949587] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.955627] LNet: Removed LNI 10.1.2.3@tcp [ 1014.958218] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.988906] LNet: Removed LNI 192.168.204.11@tcp [ 1015.031585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.117398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.149039] LNet: Added LNI 192.168.204.11@tcp [8/256/0/182] [ 1015.161034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1015.254448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.278373] LNet: 53161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.291795] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.300626] LNet: Removed LNI 10.1.2.3@tcp [ 1015.303054] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.318821] LNet: Removed LNI 192.168.204.11@tcp [ 1015.351913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.447274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.475356] LNet: Added LNI 192.168.204.11@tcp [8/256/0/182] [ 1015.484817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1015.489777] LNet: 53169:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1015.575129] LNet: Removed LNI 10.1.2.3@tcp [ 1015.587637] LNet: Removed LNI 192.168.204.11@tcp [ 1016.305259] Key type .llcrypt unregistered [ 1016.308133] Key type ._llcrypt unregistered [ 1029.845165] Key type ._llcrypt registered [ 1029.849081] Key type .llcrypt registered [ 1031.345780] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 18:28:15 (1789684095) [ 1033.006727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1034.945663] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:28:19 (1789684099) [ 1035.463292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.495875] alg: No test for adler32 (adler32-zlib) [ 1036.456942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.519970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.544415] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1037.411810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.11 [ 1037.430844] LNet: 54432:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1037.438718] LNet: Removed LNI 192.168.204.11@tcp [ 1044.673301] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:28:29 (1789684109) [ 1044.788722] LNet: 54995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.803840] LNetError: 54995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.830536] LNet: Removed LNI 192.168.204.11@tcp [ 1045.354412] Key type .llcrypt unregistered [ 1045.358765] Key type ._llcrypt unregistered [ 1051.724822] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1052.058198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.762664] Key type ._llcrypt registered [ 1054.766893] Key type .llcrypt registered [ 1054.951883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.985995] alg: No test for adler32 (adler32-zlib) [ 1056.354970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1062.439766] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:28:47 (1789684127) [ 1062.647495] LNet: 55914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.666474] LNetError: 55914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.694411] LNet: Removed LNI 10.1.2.3@tcp [ 1063.275738] Key type .llcrypt unregistered [ 1063.277757] Key type ._llcrypt unregistered [ 1069.181871] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1069.462914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1069.467530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1069.471367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1072.239521] Key type ._llcrypt registered [ 1072.244555] Key type .llcrypt registered [ 1072.408930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.427555] alg: No test for adler32 (adler32-zlib) [ 1073.400883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1080.115360] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:29:04 (1789684144) [ 1081.365733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1083.414827] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:29:07 (1789684147) [ 1083.557309] LNet: 57103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.566996] LNetError: 57103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.591708] LNet: Removed LNI 10.1.2.3@tcp [ 1084.380468] Key type .llcrypt unregistered [ 1084.386101] Key type ._llcrypt unregistered [ 1090.153476] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1090.377128] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1090.379951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1090.384994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.866423] Key type ._llcrypt registered [ 1091.869487] Key type .llcrypt registered [ 1092.074333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.098188] alg: No test for adler32 (adler32-zlib) [ 1093.220732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.330406] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1102.665490] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:29:27 (1789684167) [ 1103.395798] LNet: 58330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.403654] LNetError: 58330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1103.423600] LNet: Removed LNI 10.1.2.3@tcp [ 1104.069221] Key type .llcrypt unregistered [ 1104.071515] Key type ._llcrypt unregistered [ 1111.923395] Key type ._llcrypt registered [ 1111.927243] Key type .llcrypt registered [ 1112.251865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.280789] alg: No test for adler32 (adler32-zlib) [ 1113.192615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1113.211604] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1121.126642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1121.602066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1121.639691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1121.675952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1121.690301] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1121.892732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.923490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.942811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.973136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.998943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1122.018913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.043293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.085428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.124184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.147613] LNet: 58767:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.165524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.208217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.246806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.274362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.319188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.363451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.417233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.936607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1122.969791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1123.009140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1123.023640] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1123.205562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.244114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.293345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.319561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.341498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1123.387294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.414972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.462392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.487207] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1123.499812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.528441] LNet: 58878:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.559745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.603054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.631400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.670028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.704684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.732540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.765819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.176115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1124.196875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1124.222487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1124.236916] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1124.390923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.414164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.442793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.477484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.524877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1124.548261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.582649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.629930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.676702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.698389] LNet: 58989:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.734694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.773748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.806218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.848607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.890370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.918683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.956641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.391766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1125.418264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1125.460923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1125.684473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.712206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.745748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.780694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.828026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1125.869558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.897228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.929763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.974634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.007877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.059554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.090264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.115902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.155530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.194969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.219847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.642936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1126.711375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1126.798483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1126.837561] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1126.843770] LNet: Skipped 2 previous similar messages [ 1127.115721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.139882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.165641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.205244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.225276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1127.248605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.275613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.306692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.332445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.345218] LNet: 59212:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.349232] LNet: 59212:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1127.360989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.389056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.424228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.450964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.473504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.495647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.516330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.015923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1128.056918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1128.088962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1128.403857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.436030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.485698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.520843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.565626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1128.596646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.628809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.667794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.714455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.743393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.767988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.808931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.863166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.892195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.974088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.036142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.246943] LNet: 59364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.258951] LNetError: 59364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.286602] LNet: Removed LNI 192.168.204.11@tcp [ 1129.653982] Key type .llcrypt unregistered [ 1129.660806] Key type ._llcrypt unregistered [ 1151.359644] Key type ._llcrypt registered [ 1151.361615] Key type .llcrypt registered [ 1152.930640] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:30:17 (1789684217) [ 1153.979306] Key type .llcrypt unregistered [ 1153.984822] Key type ._llcrypt unregistered [ 1160.775380] Key type ._llcrypt registered [ 1160.779444] Key type .llcrypt registered [ 1160.938927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.951681] alg: No test for adler32 (adler32-zlib) [ 1161.929557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.975237] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1169.344667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1179.230208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1179.274935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.297840] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1180.403995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1180.436232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 1180.485944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1180.521030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 1180.556389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 1180.594813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 1180.626431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 1180.659470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 1180.697855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1180.716696] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1180.722725] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1180.952542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.979189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.013442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.054927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.092184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.121863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.156165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.199324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.228617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.242201] LNet: 60832:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.258075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.288459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.324809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.357622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.401852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.440958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.476451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.011543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1182.052657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 1182.087558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1182.121328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 1182.150322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 1182.180977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 1182.215578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 1182.273617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 1182.305279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1182.326754] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1182.331175] LNet: Skipped 2 previous similar messages [ 1182.526603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.558561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.588629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.611539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.643735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1182.681624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.729829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.779919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.806313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.824426] LNet: 60955:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.855583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.884563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.916978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.965611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.997954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.032611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.063617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.510434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1183.546676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 1183.588451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1183.633376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 1183.668466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 1183.703909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 1183.740712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 1183.798836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 1183.847814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1183.870756] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1183.875577] LNet: Skipped 2 previous similar messages [ 1184.092280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.148224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.199000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.242368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.276215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1184.318527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.350835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.380979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.419375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.436968] LNet: 61079:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.453896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.482318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.521883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.569563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.620306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.651948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.682574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.102040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1185.126488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 1185.161952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1185.201510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 1185.246508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 1185.296252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 1185.343283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 1185.386699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 1185.425372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1185.639796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.666599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.703342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.745330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.779554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1185.814201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.847217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.905472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.971411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.022529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.084763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.111839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.144356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.191362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.244803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.282050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.758519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1186.795936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 1186.830278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1186.869988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 1186.908080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 1186.957367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 1186.986777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 1187.013666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 1187.053461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1187.077773] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1187.087873] LNet: Skipped 5 previous similar messages [ 1187.293299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.324972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.375430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.407367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.440756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1187.488753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.535173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.566973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.607515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.626177] LNet: 61325:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.632080] LNet: 61325:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1187.652785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.691218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.725391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.753931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.782657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.812076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.844962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.395858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1188.445088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 1188.497745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1188.552840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 1188.616277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 1188.679127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 1188.741333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 1188.812453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 1188.847426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1189.136848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.169858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.207070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.239256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.279858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1189.318814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.349355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.386745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.408205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.437479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.470542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.521377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.546709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.575543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.611760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.643401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.588646] LNet: 61525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.597349] LNetError: 61525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.602291] LNet: Removed LNI 192.168.204.11@tcp [ 1195.625953] LNet: Removed LNI 192.168.204.11@tcp1 [ 1196.042634] Key type .llcrypt unregistered [ 1196.044495] Key type ._llcrypt unregistered [ 1212.065809] Key type ._llcrypt registered [ 1212.068535] Key type .llcrypt registered [ 1213.406052] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:31:18 (1789684278) [ 1214.455651] Key type .llcrypt unregistered [ 1214.458199] Key type ._llcrypt unregistered [ 1222.576636] Key type ._llcrypt registered [ 1222.578774] Key type .llcrypt registered [ 1222.867740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.887093] alg: No test for adler32 (adler32-zlib) [ 1223.827393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.874619] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1233.076279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1233.758311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1233.817484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1233.885275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1233.936783] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1234.266274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.302770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.335665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.362283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.383512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1234.402718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.427290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.449967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.480929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.520104] LNet: 62772:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.565424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.599168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.626654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.645064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.666378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.690614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.727224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1235.470536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1235.513601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1235.562955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1235.593256] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1235.910330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.966552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.019582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.070474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.105291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1236.153028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.197711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.223536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.256398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.274890] LNet: 62884:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.304406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.351598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.388935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.428391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.465529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.499543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.547764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.077786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1237.115668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1237.157556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1237.187568] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1237.449502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.479870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.520463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.564040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.630260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1237.691927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.749443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.787085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.851362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.885296] LNet: 62995:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.909831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.954165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.004950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.056850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.089455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.145322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.190278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.529924] LNet: 63036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.534927] LNetError: 63036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.567905] LNet: Removed LNI 192.168.204.11@tcp [ 1238.964954] Key type .llcrypt unregistered [ 1238.968935] Key type ._llcrypt unregistered [ 1255.749965] Key type ._llcrypt registered [ 1255.753520] Key type .llcrypt registered [ 1257.082824] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:32:01 (1789684321) [ 1257.845581] Key type .llcrypt unregistered [ 1257.847599] Key type ._llcrypt unregistered [ 1263.217839] Key type ._llcrypt registered [ 1263.222073] Key type .llcrypt registered [ 1263.388979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1263.407618] alg: No test for adler32 (adler32-zlib) [ 1264.334871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1264.377662] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1270.737837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1278.433818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.481414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1278.502265] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1279.749648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1279.784494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1279.824390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1279.865699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1279.900192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1279.927909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1279.963090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1280.000290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1280.043542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1280.064301] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1280.073716] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1280.276496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.297676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.331209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.370579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.406821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.453841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.495725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.545659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.584428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.608907] LNet: 64500:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.640490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.672701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.703390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.732191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.768786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.795366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.819778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.844582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.298548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1281.339112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1281.368902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1281.394170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1281.420872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1281.447735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1281.475568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1281.500122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1281.528981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1281.545346] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1281.552113] LNet: Skipped 2 previous similar messages [ 1281.730459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.756628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.786640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.827198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.854417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1281.882903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.909144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.946493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.978952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.996854] LNet: 64626:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.017668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.057821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.095207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.123931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.153721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.185707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.219261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.247658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.747897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1282.785264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1282.821805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1282.851793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1282.889607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1282.919715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1282.978684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1283.040329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1283.097087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1283.114210] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1283.120252] LNet: Skipped 1 previous similar message [ 1283.325108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.351924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.392298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.430912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.462883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1283.509749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.539691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.567620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.592651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.613473] LNet: 64751:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.636886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.682862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.717383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.754346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.798377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.838174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.883241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1283.919912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.629740] LNet: 64829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.637697] LNetError: 64829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.642890] LNet: Removed LNI 192.168.204.11@tcp [ 1288.655298] LNet: Removed LNI 192.168.204.11@tcp1 [ 1289.005717] Key type .llcrypt unregistered [ 1289.009901] Key type ._llcrypt unregistered [ 1304.594736] Key type ._llcrypt registered [ 1304.597891] Key type .llcrypt registered [ 1306.101026] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:32:50 (1789684370) [ 1307.923458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1309.885919] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:32:54 (1789684374) [ 1310.984571] Key type .llcrypt unregistered [ 1310.989192] Key type ._llcrypt unregistered [ 1318.023884] Key type ._llcrypt registered [ 1318.028925] Key type .llcrypt registered [ 1318.246853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.256965] alg: No test for adler32 (adler32-zlib) [ 1319.163930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1319.230262] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1328.239854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1328.836930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1328.858401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1328.889023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1328.916114] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1329.178233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.218301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1329.275351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.315559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1329.373042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1329.435416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1329.505953] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1329.508863] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1329.528610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1329.607661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.709643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.736397] LNet: 66371:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.783186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.861346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.911455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.961945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.981083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.000219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.272811] LNet: 66410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.279429] LNetError: 66410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.293205] LNet: Removed LNI 192.168.204.11@tcp [ 1330.703574] Key type .llcrypt unregistered [ 1330.709418] Key type ._llcrypt unregistered [ 1351.033752] Key type ._llcrypt registered [ 1351.037065] Key type .llcrypt registered [ 1351.232837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1351.248153] alg: No test for adler32 (adler32-zlib) [ 1352.120201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1352.145549] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1359.896949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1368.191713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.223774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1368.244608] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1369.375614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1369.411224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1369.454761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1369.495261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1369.531361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1369.564611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1369.605307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1369.673678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1369.726217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1369.766439] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1369.988823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.027740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1370.065809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.102793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.125488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1370.158198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.205505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.252341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.296624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.317269] LNet: 67305:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.352266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.387874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.419670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.457181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.492871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.520405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1374.681459] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1374.692794] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1374.975923] LNet: 67379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.988307] LNetError: 67379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.995276] LNet: Removed LNI 192.168.204.11@tcp [ 1375.018196] LNet: Removed LNI 192.168.204.11@tcp1 [ 1375.321438] Key type .llcrypt unregistered [ 1375.324158] Key type ._llcrypt unregistered [ 1389.392702] Key type ._llcrypt registered [ 1389.394632] Key type .llcrypt registered [ 1390.517692] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:34:15 (1789684455) [ 1390.875904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1390.896086] alg: No test for adler32 (adler32-zlib) [ 1391.826257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.875330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1391.893095] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1392.402291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1392.423735] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1393.079841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1393.147979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1393.196733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1393.231197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1393.255359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1393.279479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1393.305852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1393.322950] LNetError: 68477:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1393.887636] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1393.890170] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1395.935547] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1396.662088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1399.007361] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1400.107486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1400.157665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1400.218561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.243380] LNet: 68585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.248775] LNetError: 68585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.255754] LNet: Removed LNI 192.168.204.11@tcp [ 1400.263906] LNet: Removed LNI 192.168.204.11@tcp1 [ 1400.283928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.322591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.344719] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1400.895840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.924473] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1401.588293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1401.625747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.677859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1401.719067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1401.748833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1401.788621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.827395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.862270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1401.893902] LNetError: 68857:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1402.344841] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1405.368916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1408.480327] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1408.493030] LNet: Skipped 1 previous similar message [ 1413.340094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1413.394911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1413.427614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.232654] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:34:44 (1789684484) [ 1420.420260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.455554] LNet: 69573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.486530] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.498287] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.510978] LNet: Removed LNI 192.168.204.11@tcp [ 1420.527312] LNet: Removed LNI 192.168.204.11@tcp1 [ 1420.552532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.631354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.652901] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1421.466507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1421.488817] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1422.203429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1422.308905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1422.362861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1422.404711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1422.441622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1422.478799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1422.516941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1422.618540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1422.634117] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.11@tcp [ 1423.647977] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1423.651183] LNet: Skipped 1 previous similar message [ 1439.215937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1439.503491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1439.579847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1439.614285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 500 [ 1439.775802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1445.457579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1445.594463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.627975] LNet: 70067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.636436] LNetError: 70067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.642235] LNetError: 70067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1445.648239] LNet: Removed LNI 192.168.204.11@tcp [ 1445.656420] LNet: Removed LNI 192.168.204.11@tcp1 [ 1445.674337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.747909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.788871] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1447.159052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1447.219708] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1448.270110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1448.338206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1448.423484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1448.493452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1448.628918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1448.773638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1448.865360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1448.964506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1450.848565] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1450.866689] LNet: Skipped 3 previous similar messages [ 1451.891320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1461.409369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1461.493963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1461.539394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1469.797729] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:35:34 (1789684534) [ 1470.437328] LNet: 71114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.441396] LNetError: 71114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.447620] LNetError: 71114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1470.452564] LNet: Removed LNI 192.168.204.11@tcp [ 1470.461575] LNet: Removed LNI 192.168.204.11@tcp1 [ 1471.105407] Key type .llcrypt unregistered [ 1471.107178] Key type ._llcrypt unregistered [ 1476.593483] Key type ._llcrypt registered [ 1476.594965] Key type .llcrypt registered [ 1476.751444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1476.768105] alg: No test for adler32 (adler32-zlib) [ 1477.745329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1477.784286] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1485.471274] Lustre: DEBUG MARKER: Initial discovery [ 1485.565490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1487.394616] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1487.428857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1487.461530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1487.484283] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1487.511611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.574569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -l 3 [ 1487.625285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -e local_error [ 1487.667741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m PUT -l 6 [ 1488.034346] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1488.908573] Lustre: DEBUG MARKER: Force oleg411-client.virtnet to discover 192.168.204.111@tcp (in background) [ 1488.998768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1490.268698] Lustre: DEBUG MARKER: Force oleg411-server to discover 192.168.204.11@tcp [ 1492.064393] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1492.570786] Lustre: DEBUG MARKER: Wait for 71867 [ 1494.062760] Lustre: DEBUG MARKER: Finished wait on 71867 [ 1494.166570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1494.202466] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1494.209537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1494.427296] LNet: 72323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.435488] LNetError: 72323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.451929] LNet: Removed LNI 192.168.204.11@tcp [ 1494.908442] Key type .llcrypt unregistered [ 1494.913016] Key type ._llcrypt unregistered [ 1508.992413] Key type ._llcrypt registered [ 1508.995643] Key type .llcrypt registered [ 1509.994666] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:36:14 (1789684574) [ 1510.454455] Key type .llcrypt unregistered [ 1510.460289] Key type ._llcrypt unregistered [ 1515.371529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1515.603299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1515.610860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1515.619958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1518.072339] Key type ._llcrypt registered [ 1518.073853] Key type .llcrypt registered [ 1518.255766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1518.270562] alg: No test for adler32 (adler32-zlib) [ 1519.187460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.240206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.259475] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1519.918234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1519.945457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1527.153856] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:36:32 (1789684592) [ 1527.447480] LNet: 74396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.453974] LNetError: 74396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1527.461604] LNet: Removed LNI 10.1.2.3@tcp [ 1527.475438] LNet: Removed LNI 192.168.204.11@tcp [ 1528.055752] Key type .llcrypt unregistered [ 1528.060933] Key type ._llcrypt unregistered [ 1533.937917] Key type ._llcrypt registered [ 1533.940614] Key type .llcrypt registered [ 1534.147609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1534.163712] alg: No test for adler32 (adler32-zlib) [ 1535.087815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.140096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.175384] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1535.829129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1535.854654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1536.826795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.11@tcp [ 1537.891773] LNetError: Unexpected error -22 connecting to 192.168.204.11@tcp at host 192.168.204.11:988 [ 1542.521948] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:36:47 (1789684607) [ 1542.628515] LNet: 75518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.634166] LNetError: 75518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1543.647213] LNet: Removed LNI 10.1.2.3@tcp [ 1543.652621] LNetError: 75518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.675926] LNet: Removed LNI 192.168.204.11@tcp [ 1544.224773] Key type .llcrypt unregistered [ 1544.228909] Key type ._llcrypt unregistered [ 1549.475991] Key type ._llcrypt registered [ 1549.479083] Key type .llcrypt registered [ 1549.665482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1549.682163] alg: No test for adler32 (adler32-zlib) [ 1550.601955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.687835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.735432] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1551.473954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1551.492106] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1552.296404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.11@tcp1 --nid 192.168.204.11@tcp2 [ 1556.228633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp2 [ 1556.246560] LNetError: 76356:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.11@tcp1 to 192.168.204.11@tcp2 - Route table may be misconfigured [ 1556.276287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp2 192.168.204.11@tcp1 [ 1562.106309] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:37:06 (1789684626) [ 1562.268693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.301515] LNet: 76924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.310098] LNetError: 76924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.313853] LNet: Removed LNI 192.168.204.11@tcp1 [ 1563.368430] LNet: Removed LNI 192.168.204.11@tcp2 [ 1563.409585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.496398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.543703] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1564.264315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.284517] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1564.914873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1564.949947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.11@tcp [ 1564.977915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp1 [ 1565.013124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp -e network_timeout [ 1565.044843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -e network_timeout [ 1565.077285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -e network_timeout [ 1565.110488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -e network_timeout [ 1565.168626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1565.193404] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1565.215704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1565.311598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.415453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1565.475745] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1571.178781] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:37:16 (1789684636) [ 1571.462363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.507151] LNet: 77791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.522621] LNetError: 77791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.531744] LNetError: 77791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1571.545922] LNet: Removed LNI 192.168.204.11@tcp [ 1572.583574] LNet: Removed LNI 192.168.204.11@tcp1 [ 1572.617428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.689814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1572.985832] LNet: 77829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.375585] Key type .llcrypt unregistered [ 1573.381689] Key type ._llcrypt unregistered [ 1582.091697] Key type ._llcrypt registered [ 1582.095200] Key type .llcrypt registered [ 1583.156621] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:37:28 (1789684648) [ 1583.469100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1583.476622] alg: No test for adler32 (adler32-zlib) [ 1584.324034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.480912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1586.902266] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:37:31 (1789684651) [ 1586.983638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.013926] LNet: 78916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.037737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.168916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.205797] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1587.808442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1587.835584] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1588.620961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1588.679221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp1 [ 1588.725701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp1 [ 1593.500332] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:37:38 (1789684658) [ 1594.823566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg411-server [ 1596.176043] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:37:41 (1789684661) [ 1597.648414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg411-server [ 1599.015307] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:37:43 (1789684663) [ 1600.914361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg411-server [ 1602.309686] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:37:47 (1789684667) [ 1603.925723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg411-server [ 1605.466837] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:37:50 (1789684670) [ 1606.985373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg411-server [ 1608.450427] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:37:53 (1789684673) [ 1609.776575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg411-server [ 1611.197629] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:37:56 (1789684676) [ 1612.549108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg411-server [ 1614.046564] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:37:59 (1789684679) [ 1615.442649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg411-server [ 1616.826718] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:38:01 (1789684681) [ 1618.470289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg411-server [ 1619.803551] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:38:04 (1789684684) [ 1620.088321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.102885] LNet: 82799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.113805] LNetError: 82799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.151536] LNet: Removed LNI 192.168.204.11@tcp [ 1621.156634] LNetError: 82799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.168847] LNet: Removed LNI 192.168.204.11@tcp1 [ 1621.194638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.246640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.263767] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1621.903545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1621.968029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1622.116847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.135076] LNet: 82946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.211172] LNet: Removed LNI 192.168.204.11@tcp [ 1623.230820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.294403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1623.317121] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1623.888705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1624.049565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.068767] LNetError: 83092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.074784] LNetError: 83092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1624.093480] LNet: Removed LNI 192.168.204.11@tcp [ 1624.126606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.178997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.207496] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1624.734447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1624.785131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1624.945240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.965874] LNet: 83239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.973784] LNet: 83239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1626.026755] LNet: Removed LNI 192.168.204.11@tcp [ 1626.040988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.093269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1626.115967] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1626.727318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1626.914676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.938200] LNetError: 83384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.941914] LNetError: 83384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1626.957830] LNet: Removed LNI 192.168.204.11@tcp [ 1626.974819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.014992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.034086] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1627.669677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1627.738844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1627.869938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.902749] LNet: Removed LNI 192.168.204.11@tcp [ 1628.922956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.996607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1629.019203] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1629.610448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1629.764655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.780837] LNet: 83677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.787403] LNet: 83677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1630.822135] LNet: Removed LNI 192.168.204.11@tcp [ 1630.837188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.898676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.918665] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1631.469913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1631.517822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1631.673680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.688450] LNetError: 83824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.696330] LNetError: 83824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1631.712370] LNet: Removed LNI 192.168.204.11@tcp [ 1631.727781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.761363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1631.785864] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1632.346109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1632.531140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.573641] LNet: Removed LNI 192.168.204.11@tcp [ 1633.598913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.640160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.669227] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1634.248881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1634.308480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1634.478486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.557773] LNet: Removed LNI 192.168.204.11@tcp [ 1635.579498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.623849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1635.657508] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1636.326843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1637.684706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.758326] LNet: Removed LNI 192.168.204.11@tcp [ 1638.777175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.820690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.840386] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1639.378656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1639.430242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1641.796750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.811925] LNet: 84437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.815098] LNet: 84437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1641.819317] LNetError: 84437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.826509] LNetError: 84437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1641.840894] LNet: Removed LNI 192.168.204.11@tcp [ 1641.853953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.885489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1641.905043] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1642.356188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1643.538152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.581856] LNet: Removed LNI 192.168.204.11@tcp [ 1644.600830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.650621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.676287] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1645.259024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1645.304576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.328835] LNet: Removed LNI 192.168.204.11@tcp [ 1645.350535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.408307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.433849] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1645.914298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1645.961658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.986227] LNet: Removed LNI 192.168.204.11@tcp [ 1646.004509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.052988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1646.126113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.154377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.195958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1646.259280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.293723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.326912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1646.397960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.437772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.499814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1646.529722] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1647.128414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.155722] LNet: Removed LNI 192.168.204.11@tcp [ 1647.184557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.267257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1652.591256] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:38:37 (1789684717) [ 1652.695683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.745217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.815245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1652.839868] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1653.847826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.869544] LNet: Removed LNI 192.168.204.11@tcp231 [ 1653.891289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.961586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1653.981840] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1653.985935] LNet: 85758:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1654.289742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1654.310062] LNet: Removed LNI 192.168.204.11@tcp231 [ 1654.324221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1654.354359] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1655.209456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.230432] LNet: Removed LNI 192.168.204.11@tcp231 [ 1655.250480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.330662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1655.350426] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1655.354607] LNet: 85911:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.645902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.685079] LNet: Removed LNI 192.168.204.11@tcp231 [ 1655.706314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1655.733070] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1656.697758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.721685] LNet: Removed LNI 192.168.204.11@tcp231 [ 1656.753736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.820660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1656.851676] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1656.855714] LNet: 86062:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.182086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.213914] LNet: Removed LNI 192.168.204.11@tcp231 [ 1657.243745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1657.289289] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1658.072864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.089674] LNet: 86205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.093925] LNet: 86205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1658.099466] LNetError: 86205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.106126] LNetError: 86205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1658.120566] LNet: Removed LNI 192.168.204.11@tcp231 [ 1658.148864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.194490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1658.219557] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1658.508845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1658.545900] LNet: Removed LNI 192.168.204.11@tcp231 [ 1658.564935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1658.585469] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1664.691606] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:38:49 (1789684729) [ 1665.000864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.042071] LNet: Removed LNI 192.168.204.11@tcp231 [ 1665.063001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.136795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1665.161675] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1665.805223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1665.835849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1665.861398] LNet: Removed LNI 192.168.204.11@tcp232 [ 1665.874800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1665.895245] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1666.423380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1666.466985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1666.517337] LNet: Removed LNI 192.168.204.11@tcp232 [ 1666.534875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1666.562527] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1667.070521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1667.113546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1667.145261] LNet: Removed LNI 192.168.204.11@tcp232 [ 1667.164752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1667.182810] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1667.786995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1667.811192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1667.835808] LNet: Removed LNI 192.168.204.11@tcp232 [ 1667.850451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1672.634881] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:38:57 (1789684737) [ 1673.473198] Key type .llcrypt unregistered [ 1673.479792] Key type ._llcrypt unregistered [ 1678.036620] Key type ._llcrypt registered [ 1678.040329] Key type .llcrypt registered [ 1678.213084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1678.221695] alg: No test for adler32 (adler32-zlib) [ 1679.129450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1679.162892] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1685.061564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1690.952400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.990424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.017225] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1692.086412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1692.124128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1692.165500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1692.193734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1692.222296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1692.255820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1692.291135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1692.319898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1692.352969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1692.368925] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1692.372797] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1692.545751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.575166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.605748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.629830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.656272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1692.678680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.703596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.724138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1692.748962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1692.762782] LNet: 88712:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1692.777203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1692.798597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1692.821378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1692.841729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1692.867555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1692.890897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1693.230057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1693.259414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1693.288251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1693.319397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1693.353390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1693.387791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1693.417641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1693.450964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1693.483434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1693.507240] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1693.659964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1693.688595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1693.714424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1693.735044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1693.767487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1693.791094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.822176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.851267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.878540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.900928] LNet: 88834:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.917763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.948088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.973737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.997884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1694.032942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1694.061459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.477706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1694.519278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1694.564414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1694.606720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1694.642602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1694.674951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1694.701737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1694.732287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1694.756386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1694.776717] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1694.959734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.987581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1695.024734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1695.052718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1695.083430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1695.115718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.157551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.189466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.230893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.255483] LNet: 88956:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.270815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.301815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.329454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.355833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.378977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.399332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.787131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1695.816111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1695.847541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1695.876056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1695.905140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1695.937920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1695.966977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1695.995932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1696.018087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1696.209907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.236622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.267832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.296330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.316342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1696.338082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.357599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.375479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.394445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.424414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.446781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.467931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.490640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.511772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.530617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.887333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1696.942197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1696.967798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1696.994436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1697.034764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1697.077212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1697.125526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1697.160741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1697.185339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1697.204316] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1697.208751] LNet: Skipped 1 previous similar message [ 1697.411961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1697.445776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1697.494832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1697.540059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1697.570055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1697.605542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1697.637872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1697.676098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1697.708032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1697.723580] LNet: 89201:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1697.730229] LNet: 89201:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1697.745378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1697.768754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1697.799361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1697.839088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1697.877393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1697.910178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1698.298923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1698.329623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1698.359236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1698.397809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1698.427768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1698.463762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1698.500851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1698.546355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1698.585744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1698.615802] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1698.758165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.779721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1698.802694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.821424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.849500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1698.877613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.904810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.927402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.961525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.989763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1699.011193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1699.036854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1699.070306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1699.094745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1699.123978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.843899] LNet: 89397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.850087] LNetError: 89397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.856509] LNet: Removed LNI 192.168.204.11@tcp [ 1702.870600] LNet: Removed LNI 192.168.204.11@tcp1 [ 1703.099766] Key type .llcrypt unregistered [ 1703.102164] Key type ._llcrypt unregistered [ 1712.758212] Key type ._llcrypt registered [ 1712.760486] Key type .llcrypt registered [ 1713.517451] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:39:38 (1789684778) [ 1713.716935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.723964] alg: No test for adler32 (adler32-zlib) [ 1714.577091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.626680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.640349] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1715.168622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1715.190509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1715.210299] LNetError: 90373:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1715.264137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.11@tcp [ 1715.333377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.226428] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:39:44 (1789684784) [ 1719.341324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.356844] LNet: 90962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.361287] LNetError: 90962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.371023] LNet: Removed LNI 192.168.204.11@tcp [ 1719.388233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.422326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.437583] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1719.895421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1719.911451] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1720.521053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.11@tcp [ 1720.533099] LNetError: 91228:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1721.439394] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1721.440661] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1721.661231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.11@tcp [ 1727.600796] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:39:52 (1789684792) [ 1728.689669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg411-server [ 1729.667605] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:39:54 (1789684794) [ 1730.685823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg411-server [ 1731.635950] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:39:56 (1789684796) [ 1732.917835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg411-server [ 1733.945804] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:39:59 (1789684799) [ 1734.099928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.115167] LNet: 92858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.119668] LNetError: 92858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.123852] LNet: Removed LNI 192.168.204.11@tcp [ 1734.131393] LNet: Removed LNI 192.168.204.11@tcp1 [ 1734.142493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.197928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.225343] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1734.614523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1734.662186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1734.697190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1734.779400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1734.817801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1734.879300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1739.308310] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:40:04 (1789684804) [ 1739.378662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.401104] LNet: 93582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.407977] LNetError: 93582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.414375] LNetError: 93582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1739.426847] LNet: Removed LNI 192.168.204.11@tcp [ 1739.445683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.502748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.537867] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1744.316892] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:40:09 (1789684809) [ 1744.540746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.558469] LNet: 94291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.565530] LNetError: 94291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.580804] LNet: Removed LNI 192.168.204.11@tcp [ 1744.596835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.642978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.682693] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1749.163982] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:40:14 (1789684814) [ 1750.137246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1751.215309] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:40:16 (1789684816) [ 1751.561938] LNet: 95326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.565955] LNetError: 95326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.579822] LNet: Removed LNI 192.168.204.11@tcp [ 1752.031837] Key type .llcrypt unregistered [ 1752.034465] Key type ._llcrypt unregistered [ 1755.990959] Key type ._llcrypt registered [ 1755.993256] Key type .llcrypt registered [ 1756.121739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1756.134320] alg: No test for adler32 (adler32-zlib) [ 1757.011480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1757.031615] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1761.690282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1764.528165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.111@tcp [ 1764.744327] LNet: 95739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.749847] LNetError: 95739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.757577] LNet: Removed LNI 192.168.204.11@tcp [ 1764.985577] Key type .llcrypt unregistered [ 1764.987100] Key type ._llcrypt unregistered [ 1772.106657] Key type ._llcrypt registered [ 1772.108902] Key type .llcrypt registered [ 1773.222407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1774.088688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1775.214614] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:40:40 (1789684840) [ 1776.371326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1777.438087] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:40:42 (1789684842) [ 1778.786796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg411-server [ 1780.075601] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:40:45 (1789684845) [ 1781.358715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg411-server [ 1782.613702] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:40:47 (1789684847) [ 1783.464934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.575401] alg: No test for adler32 (adler32-zlib) [ 1784.559797] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1784.750338] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1786.391435] Key type lgssc registered [ 1787.315341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1800.373836] Key type lgssc unregistered [ 1800.526264] LNet: 98906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.533188] LNetError: 98906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.545419] LNet: Removed LNI 192.168.204.11@tcp [ 1800.947180] Key type .llcrypt unregistered [ 1800.952918] Key type ._llcrypt unregistered [ 1808.537537] Key type ._llcrypt registered [ 1808.539320] Key type .llcrypt registered [ 1809.319235] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:41:14 (1789684874) [ 1809.632142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.646078] alg: No test for adler32 (adler32-zlib) [ 1810.511580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.556192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1810.589232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1810.629707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1810.656355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1814.524465] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:41:19 (1789684879) [ 1814.583615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.598538] LNet: 100310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.615509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.663869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1814.679753] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1815.172873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.185600] LNet: 100460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.190712] LNetError: 100460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.205857] LNet: Removed LNI 192.168.204.11@tcp [ 1815.219673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.257211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1815.271828] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1815.520572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.548985] LNet: Removed LNI 192.168.204.11@tcp [ 1815.563318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.594343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1815.611801] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1818.763308] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:41:24 (1789684884) [ 1818.865807] LNet: 101079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.869430] LNet: 101079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1818.873191] LNetError: 101079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.876577] LNetError: 101079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1818.891239] LNet: Removed LNI 192.168.204.11@tcp [ 1819.191382] Key type .llcrypt unregistered [ 1819.193289] Key type ._llcrypt unregistered [ 1822.076843] Key type ._llcrypt registered [ 1822.079878] Key type .llcrypt registered [ 1822.185850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.198751] alg: No test for adler32 (adler32-zlib) [ 1823.031191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.069195] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1823.071375] LNet: *** cfs_fail_loc=e003, val=100*** [ 1823.487848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1823.504627] Ping reply received with mapping: [ 1823.507911] found metadata reply NI[192.168.204.11@tcp] [ 1823.510874] 0b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1823.515709] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1823.518547] 5a 5a 5a 5a 5a 5a 5a 5a [ 1823.581133] LNet: 101473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.585792] LNetError: 101473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.593353] LNet: Removed LNI 192.168.204.11@tcp [ 1823.902557] Key type .llcrypt unregistered [ 1823.904781] Key type ._llcrypt unregistered [ 1829.196761] Key type ._llcrypt registered [ 1829.198498] Key type .llcrypt registered [ 1829.896550] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:41:35 (1789684895) [ 1830.183297] Key type .llcrypt unregistered [ 1830.185077] Key type ._llcrypt unregistered [ 1833.317277] Key type ._llcrypt registered [ 1833.318500] Key type .llcrypt registered [ 1833.414554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1833.425435] alg: No test for adler32 (adler32-zlib) [ 1834.352133] LNetError: 102509:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1834.546624] Key type .llcrypt unregistered [ 1834.548281] Key type ._llcrypt unregistered [ 1837.020599] Key type ._llcrypt registered [ 1837.022266] Key type .llcrypt registered [ 1837.745222] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:41:43 (1789684903) [ 1838.062583] Key type .llcrypt unregistered [ 1838.064520] Key type ._llcrypt unregistered [ 1841.414629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1841.420170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1843.719955] Key type ._llcrypt registered [ 1843.721599] Key type .llcrypt registered [ 1843.824277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.832572] alg: No test for adler32 (adler32-zlib) [ 1844.698063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.740698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1850.656141] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:41:55 (1789684915) [ 1850.809575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.822845] LNet: 104280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.827328] LNetError: 104280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1850.844259] LNet: Removed LNI 10.1.2.3@tcp [ 1850.857335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.281805] Key type .llcrypt unregistered [ 1851.286553] Key type ._llcrypt unregistered [ 1854.536769] Key type ._llcrypt registered [ 1854.538257] Key type .llcrypt registered [ 1854.636662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1854.642923] alg: No test for adler32 (adler32-zlib) [ 1855.490331] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1855.536330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1855.636949] LNet: 104565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.640899] LNetError: 104565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.647389] LNet: Removed LNI 192.168.204.11@tcp [ 1855.984851] Key type .llcrypt unregistered [ 1855.988233] Key type ._llcrypt unregistered [ 1861.744393] Key type ._llcrypt registered [ 1861.745957] Key type .llcrypt registered [ 1862.479291] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:42:07 (1789684927) [ 1862.769070] Key type .llcrypt unregistered [ 1862.770552] Key type ._llcrypt unregistered [ 1866.187121] Key type ._llcrypt registered [ 1866.188200] Key type .llcrypt registered [ 1866.291445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1866.305262] alg: No test for adler32 (adler32-zlib) [ 1871.515795] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:42:16 (1789684936) [ 1871.594082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.641632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.699143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.730259] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1872.259803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.290416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1872.323194] LNet: Removed LNI 192.168.204.11@tcp [ 1872.516742] LNet: 106303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.845983] Key type .llcrypt unregistered [ 1872.848536] Key type ._llcrypt unregistered [ 1879.791875] Key type ._llcrypt registered [ 1879.793878] Key type .llcrypt registered [ 1880.688790] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:42:25 (1789684945) [ 1880.919821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1880.931924] alg: No test for adler32 (adler32-zlib) [ 1881.801894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.840589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.854079] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1882.367403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1882.396737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.11@tcp [ 1882.417738] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1882.420597] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1882.423034] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1882, 0, 0 [ 1882.427039] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1882.429393] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1882.432038] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1882.435269] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1882, 0, 0 [ 1882.440276] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1882.442749] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1882.445443] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1882.447989] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1882, 0, 1932 [ 1882.452539] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1882.454702] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1882.457924] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1882.460594] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1882, 0, 1932 [ 1882.464978] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1886.403243] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:42:31 (1789684951) [ 1886.740776] LNet: 107838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.753817] LNetError: 107838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.776704] LNet: Removed LNI 192.168.204.11@tcp [ 1887.180169] Key type .llcrypt unregistered [ 1887.183964] Key type ._llcrypt unregistered [ 1890.685838] Key type ._llcrypt registered [ 1890.689285] Key type .llcrypt registered [ 1890.836538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.848426] alg: No test for adler32 (adler32-zlib) [ 1891.709892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1891.733086] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1896.481362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1901.999138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.024496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1902.038706] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1902.630811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1902.751625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1902.754168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1902.757070] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1904.963742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1904.981612] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1910.930292] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:42:56 (1789684976) [ 1911.063436] LNet: 109337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.068242] LNetError: 109337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.072408] LNet: Removed LNI 192.168.204.11@tcp [ 1912.095144] LNet: Removed LNI 192.168.204.11@tcp1 [ 1912.097207] LNetError: 109337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1912.101054] LNetError: 109337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1912.121350] LNet: Removed LNI 10.1.2.3@tcp99 [ 1912.531629] Key type .llcrypt unregistered [ 1912.533382] Key type ._llcrypt unregistered [ 1915.857107] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1915.860582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1918.170376] Key type ._llcrypt registered [ 1918.173635] Key type .llcrypt registered [ 1918.284511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1918.293567] alg: No test for adler32 (adler32-zlib) [ 1919.137594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.175889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.192776] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1919.607958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1919.623981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1920.674930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1920.698205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.720944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1920.731680] LNetError: 110077:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1920.747192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.773291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1920.948372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.978348] LNet: 110097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.981394] LNetError: 110097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1920.985496] LNet: Removed LNI 10.1.2.3@tcp [ 1922.021785] LNet: Removed LNI 192.168.204.11@tcp [ 1922.316550] Key type .llcrypt unregistered [ 1922.317949] Key type ._llcrypt unregistered [ 1927.278167] Key type ._llcrypt registered [ 1927.279570] Key type .llcrypt registered [ 1927.964055] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:43:13 (1789684993) [ 1928.128967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.136854] alg: No test for adler32 (adler32-zlib) [ 1928.984893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.012620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.026362] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1929.490398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1929.537911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg411-client@tcp [ 1932.984568] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:43:18 (1789684998) [ 1933.040437] LNet: 111619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.044789] LNetError: 111619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.059951] LNet: Removed LNI 192.168.204.11@tcp [ 1933.409660] Key type .llcrypt unregistered [ 1933.413105] Key type ._llcrypt unregistered [ 1936.701435] Key type ._llcrypt registered [ 1936.702972] Key type .llcrypt registered [ 1936.806453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.815637] alg: No test for adler32 (adler32-zlib) [ 1937.642343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1937.667432] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1937.691840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1941.096560] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:43:26 (1789685006) [ 1941.839704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1942.707765] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:43:28 (1789685008) [ 1943.556651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1944.415092] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:43:29 (1789685009) [ 1944.473794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.493440] LNet: 113036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.500354] LNetError: 113036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.509362] LNet: Removed LNI 192.168.204.11@tcp [ 1944.525830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.560330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.576585] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1944.990363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1945.016363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1948.619693] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:43:33 (1789685013) [ 1948.762152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.778239] LNet: 113752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.783526] LNetError: 113752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.799036] LNet: Removed LNI 192.168.204.11@tcp [ 1948.813518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.850065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1948.873832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1949.305267] Key type .llcrypt unregistered [ 1949.308128] Key type ._llcrypt unregistered [ 1955.825512] Key type ._llcrypt registered [ 1955.828909] Key type .llcrypt registered [ 1956.697627] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:43:41 (1789685021) [ 1956.987275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1956.996443] alg: No test for adler32 (adler32-zlib) [ 1957.857832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.892778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.921695] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1958.468572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1958.537178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1962.539306] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:43:47 (1789685027) [ 1962.669436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.688941] LNet: 115308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.693449] LNetError: 115308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.705642] LNet: Removed LNI 192.168.204.11@tcp [ 1962.724501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.768866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1962.794929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1966.648574] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:43:51 (1789685031) [ 1966.838425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.858722] LNet: 115887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.879245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.946045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1967.246678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1967.278731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1967.317739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1967.592255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1967.617495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1967.674992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.711745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.779643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1968.053796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.080543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1968.132841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1968.451787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.491938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1972.436393] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:43:57 (1789685037) [ 1972.551903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.570739] LNet: 116520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.573430] LNet: 116520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1972.588928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.639962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.663853] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1973.140784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1973.159976] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1973.728476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1973.784488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1973.953343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.972190] LNetError: 116800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.978774] LNet: Removed LNI 192.168.204.11@tcp [ 1973.998081] LNet: Removed LNI 192.168.204.11@tcp2 [ 1974.017879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.064661] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1974.070521] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1974.076043] LNet: 116806:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1974.348376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1974.472522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.519166] LNet: Removed LNI 192.168.204.11@tcp [ 1975.521743] LNetError: 116843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.525113] LNetError: 116843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1975.534921] LNet: Removed LNI 192.168.204.11@tcp2 [ 1975.550772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.585558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.599590] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1975.961296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1975.975614] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1976.429913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1976.460666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1976.504607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1976.647086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.662655] LNet: 117127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.666410] LNet: 117127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1977.695192] LNet: Removed LNI 192.168.204.11@tcp [ 1977.697034] LNetError: 117127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.704471] LNetError: 117127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1977.714653] LNet: Removed LNI 192.168.204.11@tcp2 [ 1977.731835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.785214] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1977.792818] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1977.796996] LNet: 117133:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1978.073393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1978.205236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.222955] LNet: Removed LNI 192.168.204.11@tcp [ 1979.239503] LNet: Removed LNI 192.168.204.11@tcp2 [ 1979.257746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.296692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.309478] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1979.739913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1979.755735] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1980.248516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1980.292799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1980.410939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.471147] LNet: Removed LNI 192.168.204.11@tcp [ 1981.481450] LNet: Removed LNI 192.168.204.11@tcp2 [ 1981.494086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.532748] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1981.538959] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1981.546806] LNet: 117456:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1981.815039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1981.926955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.942733] LNet: 117493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.947765] LNet: 117493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1981.953721] LNetError: 117493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.959312] LNetError: 117493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1981.964714] LNet: Removed LNI 192.168.204.11@tcp [ 1981.973423] LNet: Removed LNI 192.168.204.11@tcp2 [ 1981.993977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.032782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.053630] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1982.489319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1982.502491] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1983.067967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1983.100205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1983.131595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1983.270223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.351282] LNet: Removed LNI 192.168.204.11@tcp [ 1985.383895] LNet: Removed LNI 192.168.204.11@tcp2 [ 1985.403898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.452246] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1985.457809] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1985.462560] LNet: 117783:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1985.743907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1985.887219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.900828] LNet: Removed LNI 192.168.204.11@tcp [ 1986.920327] LNet: Removed LNI 192.168.204.11@tcp2 [ 1986.936827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.976750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1986.990838] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1987.479203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1987.491834] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1988.060408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1988.083823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1988.133763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1988.331596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.407159] LNet: Removed LNI 192.168.204.11@tcp [ 1989.421177] LNet: Removed LNI 192.168.204.11@tcp2 [ 1989.443855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.483445] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1989.487825] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1989.491873] LNet: 118123:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1989.814442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1989.996808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.015071] LNet: 118175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.025589] LNet: 118175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1990.029599] LNetError: 118175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.039778] LNetError: 118175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1990.045869] LNet: Removed LNI 192.168.204.11@tcp [ 1990.056694] LNet: Removed LNI 192.168.204.11@tcp2 [ 1990.072417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.126478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.145989] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1990.623802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1990.640702] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1991.315345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1991.342953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1991.364165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1991.408381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1991.663811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.735264] LNet: Removed LNI 192.168.204.11@tcp [ 1993.766975] LNet: Removed LNI 192.168.204.11@tcp2 [ 1993.792208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.835881] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1993.841306] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1994.162858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1998.664992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1999.608075] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:44:24 (1789685064) [ 1999.742808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1999.754689] LNet: Removed LNI 192.168.204.11@tcp [ 2000.806822] LNet: Removed LNI 192.168.204.11@tcp2 [ 2000.828398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.010259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.023772] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2001.429653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2001.449363] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2001.987186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.009827] LNet: Removed LNI 10.1.2.3@tcp [ 2002.020557] LNet: Removed LNI 192.168.204.11@tcp [ 2002.041138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.087947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2002.105585] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2002.108819] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2002.345763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2002.361393] LNet: Removed LNI 192.168.204.11@tcp [ 2002.375593] LNet: Removed LNI 10.1.2.3@tcp [ 2006.022386] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:44:31 (1789685071) [ 2006.093502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.110652] LNet: 120110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.116656] LNet: 120110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2006.133721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.182599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2006.199501] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2006.644859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2006.676442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2006.733837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.770502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.803176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.832923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.867783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.894470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.922595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.954054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2006.979080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2007.005789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2007.145945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2011.027898] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 18:44:36 (1789685076) [ 2011.137726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2011.160704] LNetError: 120856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.165961] LNetError: 120856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2011.177353] LNet: Removed LNI 192.168.204.11@tcp [ 2011.207380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.245860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2011.267034] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2011.739522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2011.757996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2011.786530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.808537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.830482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.858679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.884796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.909328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.932291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.954432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2011.978190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.007763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.075187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2012.097269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.122894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.161662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.191453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.215080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.238293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.262433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.290369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.314831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2012.346808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2015.772506] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:44:41 (1789685081) [ 2015.830887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.883347] LNet: Removed LNI 192.168.204.11@tcp [ 2016.906259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.984125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2017.108950] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2017.113445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2017.117091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2019.300470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.321347] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2019.791242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2019.806683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2021.109133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2021.130984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2021.172349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.11@tcp -m PUT -e local_timeout -r 1 [ 2021.208075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2021.251158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2021.254539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2021.356315] LNetError: 121678:0:(lib-move.c:4794:lnet_parse()) 192.168.204.11@tcp, src 192.168.204.11@tcp: Dropping PUT (error -108 looking up sender) [ 2022.367764] LNet: Removed LNI 10.1.2.3@tcp [ 2023.403312] LNet: Removed LNI 192.168.204.11@tcp [ 2023.652466] Key type .llcrypt unregistered [ 2023.654657] Key type ._llcrypt unregistered [ 2030.624140] Key type ._llcrypt registered [ 2030.625889] Key type .llcrypt registered [ 2031.664977] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:44:56 (1789685096) [ 2031.887889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.903478] alg: No test for adler32 (adler32-zlib) [ 2032.768746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.015738] LNet: 122985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.422520] Key type .llcrypt unregistered [ 2033.424823] Key type ._llcrypt unregistered [ 2037.189591] Key type ._llcrypt registered [ 2037.191696] Key type .llcrypt registered [ 2037.311832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2037.323078] alg: No test for adler32 (adler32-zlib) [ 2038.218479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2038.237685] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2042.574144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2042.758296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2043.150360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2043.313466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2043.350150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2043.359147] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 2043.383413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2043.407928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2043.432344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2043.463208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2043.483836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2043.508663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.533081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2043.548316] LNet: 123434:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2043.563882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2043.586554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.610312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.629629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.650862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.669164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2043.955381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2044.070401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2044.087555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2044.105395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2044.124746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2044.143763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2044.160377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2044.181942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2044.204895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2044.228844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2044.239379] LNet: 123553:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2044.253539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2044.287686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2044.313485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2044.336684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2044.361765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2044.384629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2044.724857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2044.857489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2044.875476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2044.898646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2044.917766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2044.938889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2044.966705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2044.985393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2045.000747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2045.019551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2045.046904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2045.073417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2045.095901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2045.120774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2045.142894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2045.166416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2045.545598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2045.669941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2045.692880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2045.717433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2045.738749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2045.766134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2045.797553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2045.826636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2045.852716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2045.876155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2045.889548] LNet: 123791:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2045.892841] LNet: 123791:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2045.903281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2045.924767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2045.950268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2045.968866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2046.001366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2046.031768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2046.209799] LNet: 123837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.214641] LNetError: 123837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.230971] LNet: Removed LNI 192.168.204.11@tcp [ 2046.471698] Key type .llcrypt unregistered [ 2046.473574] Key type ._llcrypt unregistered [ 2055.323990] Key type ._llcrypt registered [ 2055.326566] Key type .llcrypt registered [ 2056.124526] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:45:21 (1789685121) [ 2056.499501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.509260] alg: No test for adler32 (adler32-zlib) [ 2057.375343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.718403] LNet: 124720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.205530] Key type .llcrypt unregistered [ 2058.211889] Key type ._llcrypt unregistered [ 2063.125939] Key type ._llcrypt registered [ 2063.129195] Key type .llcrypt registered [ 2063.303183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.321924] alg: No test for adler32 (adler32-zlib) [ 2064.228987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2064.251734] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2069.540808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2075.113626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.132083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2075.147707] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 2075.678341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2076.165554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2076.330558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2076.347772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2076.367542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2076.386982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2076.407188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2076.431909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2076.452203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2076.475496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2076.495614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2076.513105] LNet: 125371:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2076.530584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2076.583304] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 2076.586292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2076.619619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2076.663610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2076.700559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2076.731357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2077.144678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2077.291387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2077.309595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2077.333247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2077.353554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2077.376322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2077.398982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2077.422963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2077.441688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2077.465387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2077.477663] LNet: 125490:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2077.490312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2077.510110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2077.538428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2077.561269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2077.590470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2077.613518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2077.954857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2078.094953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2078.113166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2078.135695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2078.155564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2078.176970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2078.214280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2078.243502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2078.272556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2078.301645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2078.320615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2078.342588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2078.371524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2078.392750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2078.421344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2078.452841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2078.877439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2079.041389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2079.070624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2079.091937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2079.126655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2079.149980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2079.177759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2079.214800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2079.237197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2079.268233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2079.279638] LNet: 125729:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2079.282937] LNet: 125729:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2079.294670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2079.324587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2079.348704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2079.380753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2079.403548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2079.424578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2082.428965] LNet: 125809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.433606] LNetError: 125809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.439891] LNet: Removed LNI 192.168.204.11@tcp [ 2082.446955] LNet: Removed LNI 192.168.204.11@tcp1 [ 2082.683635] Key type .llcrypt unregistered [ 2082.685301] Key type ._llcrypt unregistered [ 2091.859524] Key type ._llcrypt registered [ 2091.862350] Key type .llcrypt registered [ 2092.772905] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:45:57 (1789685157) [ 2093.249881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.262714] alg: No test for adler32 (adler32-zlib) [ 2094.149482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.229840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2094.256877] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2094.934586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2094.957679] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2095.768870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2095.833460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2096.864669] LNet: Removed LNI 192.168.204.11@tcp [ 2101.169594] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:46:06 (1789685166) [ 2102.146538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2103.153606] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:46:08 (1789685168) [ 2107.745956] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:46:13 (1789685173) [ 2112.121113] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:46:17 (1789685177) [ 2116.024862] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:46:21 (1789685181) [ 2120.406231] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:46:25 (1789685185) [ 2124.753631] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:46:30 (1789685190) [ 2129.151605] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:46:34 (1789685194) [ 2133.749122] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:46:39 (1789685199) [ 2138.281782] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:46:43 (1789685203) [ 2142.522279] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:46:47 (1789685207) [ 2146.965136] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:46:52 (1789685212) [ 2151.879785] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:46:56 (1789685216) [ 2158.397196] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:47:02 (1789685222) [ 2172.680667] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:47:15 (1789685235) [ 2187.007584] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:47:30 (1789685250) [ 2198.702730] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:47:43 (1789685263) [ 2205.080857] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:47:50 (1789685270) [ 2213.207066] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:47:57 (1789685277) [ 2213.326662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.346875] LNet: 137491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.356743] LNetError: 137491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.395131] LNet: Removed LNI 192.168.204.11@tcp2 [ 2213.451933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.593338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2213.619058] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2214.969173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.987234] LNet: 137640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.998587] LNetError: 137640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.012711] LNet: Removed LNI 192.168.204.11@tcp [ 2215.038308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.095422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2215.115209] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2222.282396] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:48:06 (1789685286) [ 2230.281966] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:48:14 (1789685294) [ 2239.167432] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:48:23 (1789685303) [ 2246.965748] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:48:31 (1789685311) [ 2253.634205] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:48:38 (1789685318) [ 2259.995668] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:48:44 (1789685324) [ 2266.060836] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:48:51 (1789685331) [ 2271.205597] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:48:56 (1789685336) [ 2277.095673] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:49:01 (1789685341) [ 2283.484416] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:49:08 (1789685348) [ 2289.031552] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:49:13 (1789685353) [ 2295.017367] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:49:20 (1789685360) [ 2300.926298] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2022 sec ========== 18:49:25 (1789685365) [ 2301.234916] LNet: 145103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.241704] LNetError: 145103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.252151] LNet: Removed LNI 192.168.204.11@tcp [ 2301.918818] Key type .llcrypt unregistered [ 2301.923069] Key type ._llcrypt unregistered [ 2311.861051] Key type ._llcrypt registered [ 2311.862430] Key type .llcrypt registered [ 2312.055934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2312.072115] alg: No test for adler32 (adler32-zlib) [ 2313.092258] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2313.296764] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2314.959345] Key type lgssc registered [ 2316.006828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.010622] Lustre: Mounted lustre-client - version 2.17.58_39_g12a437a [ 2373.934539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2379.981187] Lustre: Unmounted lustre-client [ 2401.273724] Key type lgssc unregistered [ 2401.584117] LNet: 148051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.607015] LNetError: 148051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.636238] LNet: Removed LNI 192.168.204.11@tcp [ 2402.355153] Key type .llcrypt unregistered [ 2402.358593] Key type ._llcrypt unregistered