[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472446205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002382] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007298] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012061] *** VALIDATE selinux *** [ 0.021289] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026540] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.031449] *** VALIDATE proc *** [ 0.032230] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036067] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042189] debug: unmapping init [mem 0xffffffff9de59000-0xffffffff9de60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053283] rcu: Hierarchical SRCU implementation. [ 0.055361] smp: Bringing up secondary CPUs ... [ 0.056565] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.065100] smp: Brought up 1 node, 4 CPUs [ 0.067017] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107514] node 0 deferred pages initialised in 37ms [ 0.111215] devtmpfs: initialized [ 0.112218] x86/mm: Memory block size: 128MB [ 0.115000] gcov: version magic: 0x41383552 [ 0.118220] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123246] pinctrl core: initialized pinctrl subsystem [ 0.125152] [ 0.125651] ************************************************************* [ 0.129017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131010] ** ** [ 0.134014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137013] ** ** [ 0.140013] ** This means that this kernel is built to expose internal ** [ 0.143014] ** IOMMU data structures, which may compromise security on ** [ 0.146013] ** your system. ** [ 0.151017] ** ** [ 0.153009] ** If you see this message and you are not debugging the ** [ 0.155012] ** kernel, report this immediately to your vendor! ** [ 0.157013] ** ** [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162292] ************************************************************* [ 0.165759] NET: Registered protocol family 16 [ 0.167501] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174101] cpuidle: using governor menu [ 0.176581] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180446] PCI: Using configuration type 1 for base access [ 0.183121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198196] cryptd: max_cpu_qlen set to 1000 [ 0.201294] ACPI: Added _OSI(Module Device) [ 0.202014] ACPI: Added _OSI(Processor Device) [ 0.204012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209251] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215499] ACPI: Interpreter enabled [ 0.217063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218010] ACPI: Using IOAPIC for interrupt routing [ 0.220124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236848] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.251598] acpiphp: Slot [2] registered [ 0.253114] acpiphp: Slot [5] registered [ 0.255103] acpiphp: Slot [6] registered [ 0.256186] acpiphp: Slot [3] registered [ 0.258111] acpiphp: Slot [4] registered [ 0.259187] acpiphp: Slot [7] registered [ 0.261120] acpiphp: Slot [8] registered [ 0.262079] acpiphp: Slot [9] registered [ 0.264186] acpiphp: Slot [10] registered [ 0.265096] acpiphp: Slot [11] registered [ 0.267117] acpiphp: Slot [12] registered [ 0.269110] acpiphp: Slot [13] registered [ 0.271121] acpiphp: Slot [14] registered [ 0.272093] acpiphp: Slot [15] registered [ 0.273088] acpiphp: Slot [16] registered [ 0.275092] acpiphp: Slot [17] registered [ 0.276086] acpiphp: Slot [18] registered [ 0.278131] acpiphp: Slot [19] registered [ 0.280093] acpiphp: Slot [20] registered [ 0.281071] acpiphp: Slot [21] registered [ 0.283084] acpiphp: Slot [22] registered [ 0.284084] acpiphp: Slot [23] registered [ 0.286090] acpiphp: Slot [24] registered [ 0.287116] acpiphp: Slot [25] registered [ 0.289149] acpiphp: Slot [26] registered [ 0.290096] acpiphp: Slot [27] registered [ 0.292158] acpiphp: Slot [28] registered [ 0.294188] acpiphp: Slot [29] registered [ 0.296174] acpiphp: Slot [30] registered [ 0.297153] acpiphp: Slot [31] registered [ 0.299072] PCI host bridge to bus 0000:00 [ 0.300000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.316082] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328789] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.334588] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339124] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.347018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350289] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357881] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.373016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378703] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386511] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.401013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.435413] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.500666] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.505405] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.509876] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.513386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.517234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.524013] iommu: Default domain type: Passthrough [ 0.526712] SCSI subsystem initialized [ 0.529159] ACPI: bus type USB registered [ 0.532126] usbcore: registered new interface driver usbfs [ 0.535086] usbcore: registered new interface driver hub [ 0.537098] usbcore: registered new device driver usb [ 0.539897] pps_core: LinuxPPS API ver. 1 registered [ 0.542012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.546210] PTP clock support registered [ 0.549580] EDAC MC: Ver: 3.0.0 [ 0.551578] PCI: Using ACPI for IRQ routing [ 0.553969] NetLabel: Initializing [ 0.556016] NetLabel: domain hash size = 128 [ 0.558008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.560104] NetLabel: unlabeled traffic allowed by default [ 0.562119] vgaarb: loaded [ 0.564302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.566012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.572767] clocksource: Switched to clocksource kvm-clock [ 0.693442] VFS: Disk quotas dquot_6.6.0 [ 0.695466] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.700353] *** VALIDATE ramfs *** [ 0.702672] *** VALIDATE hugetlbfs *** [ 0.706154] pnp: PnP ACPI init [ 0.710558] pnp: PnP ACPI: found 6 devices [ 0.731558] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.735870] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.738790] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.741614] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.744549] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.747379] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.751029] NET: Registered protocol family 2 [ 0.753584] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.759521] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.763605] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.774036] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.781678] TCP: Hash tables configured (established 65536 bind 65536) [ 0.786307] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.791664] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794708] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.799121] NET: Registered protocol family 1 [ 0.802805] RPC: Registered named UNIX socket transport module. [ 0.805760] RPC: Registered udp transport module. [ 0.808428] RPC: Registered tcp transport module. [ 0.810981] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.814579] NET: Registered protocol family 44 [ 0.816207] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.819433] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823225] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827200] PCI: CLS 0 bytes, default 64 [ 0.830150] Unpacking initramfs... [ 2.413542] debug: unmapping init [mem 0xffff95e27cc64000-0xffff95e27ffcffff] [ 2.417756] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.420081] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.423145] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.956387] Initialise system trusted keyrings [ 2.958522] Key type blacklist registered [ 2.960408] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.969864] zbud: loaded [ 2.975138] *** VALIDATE nfs *** [ 2.975988] *** VALIDATE nfs4 *** [ 2.979293] pstore: using deflate compression [ 2.985442] Platform Keyring initialized [ 3.112867] NET: Registered protocol family 38 [ 3.114842] Key type asymmetric registered [ 3.116355] Asymmetric key parser 'x509' registered [ 3.122839] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.125757] io scheduler mq-deadline registered [ 3.127389] io scheduler kyber registered [ 3.129983] io scheduler bfq registered [ 3.134653] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.138062] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.140752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.143682] ACPI: Power Button [PWRF] [ 3.150774] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.160089] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.180437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.211442] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.246128] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.253949] Non-volatile memory driver v1.3 [ 3.255961] Linux agpgart interface v0.103 [ 3.295691] virtio_blk virtio1: [vda] 146208 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.300346] vda: detected capacity change from 0 to 74858496 [ 3.321641] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.324606] vdb: detected capacity change from 0 to 1073741824 [ 3.335858] libphy: Fixed MDIO Bus: probed [ 3.342391] usbcore: registered new interface driver usbserial_generic [ 3.344917] usbserial: USB Serial support registered for generic [ 3.347592] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.352770] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.355291] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.358976] mousedev: PS/2 mouse device common for all mice [ 3.362239] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.366331] rtc_cmos 00:05: RTC can wake from S4 [ 3.371063] rtc_cmos 00:05: registered as rtc0 [ 3.372358] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.373855] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.373892] intel_pstate: CPU model not supported [ 3.378263] hid: raw HID events driver (C) Jiri Kosina [ 3.388044] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.388633] usbcore: registered new interface driver usbhid [ 3.401788] usbhid: USB HID core driver [ 3.403444] drop_monitor: Initializing network drop monitor service [ 3.406703] Initializing XFRM netlink socket [ 3.409686] NET: Registered protocol family 10 [ 3.413081] Segment Routing with IPv6 [ 3.414468] NET: Registered protocol family 17 [ 3.416615] mpls_gso: MPLS GSO support [ 3.424317] RAS: Correctable Errors collector initialized. [ 3.426387] AVX version of gcm_enc/dec engaged. [ 3.428092] AES CTR mode by8 optimization enabled [ 3.522473] sched_clock: Marking stable (3522427043, 0)->(4478312589, -955885546) [ 3.525869] registered taskstats version 1 [ 3.528856] Loading compiled-in X.509 certificates [ 3.532629] zswap: loaded using pool lzo/zbud [ 3.569518] Key type big_key registered [ 3.584712] Key type encrypted registered [ 3.586616] ima: No TPM chip found, activating TPM-bypass! [ 3.589218] ima: Allocated hash algorithm: sha1 [ 3.590986] ima: No architecture policies found [ 3.592987] evm: Initialising EVM extended attributes: [ 3.595170] evm: security.selinux [ 3.596549] evm: security.ima [ 3.598315] evm: security.capability [ 3.600573] evm: HMAC attrs: 0x1 [ 3.603611] rtc_cmos 00:05: setting system clock to 2026-08-24 05:15:02 UTC (1787548502) [ 3.612553] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.616818] debug: unmapping init [mem 0xffffffff9db82000-0xffffffff9de58fff] [ 3.625284] Write protecting the kernel read-only data: 28672k [ 3.631696] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 3.634544] debug: unmapping init [mem 0xffffffff9cb14000-0xffffffff9cbfffff] [ 3.677563] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.689338] systemd[1]: Detected virtualization kvm. [ 3.691472] systemd[1]: Detected architecture x86-64. [ 3.695911] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.724509] systemd[1]: No hostname configured. [ 3.728298] systemd[1]: Set hostname to . [ 3.730486] random: systemd: uninitialized urandom read (16 bytes read) [ 3.732982] systemd[1]: Initializing machine ID from random generator. [ 3.805830] random: ln: uninitialized urandom read (6 bytes read) [ 3.951165] random: systemd: uninitialized urandom read (16 bytes read) [ 3.958279] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.971711] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.976324] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.993667] device-mapper: uevent: version 1.0.3 [ 4.996795] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 5.495825] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.070214] virtio_net virtio0 ens2: renamed from eth0 [ 6.320333] scsi host0: ata_piix [ 6.420175] scsi host1: ata_piix [ 6.425786] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.432638] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.116927] random: crng init done [ 11.119289] random: 7 urandom warning(s) missed due to ratelimiting [ 12.049690] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.654709] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.454684] printk: systemd: 24 output lines suppressed due to ratelimiting [ 15.841250] SELinux: Disabled at runtime. [ 15.900090] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.909235] systemd[1]: Detected virtualization kvm. [ 15.910983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.585625] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.589452] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.595363] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.602496] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.609595] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.622978] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.638708] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.tar[ 16.830255] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS get. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.627993] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.298244] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.372732] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.614709] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.668863] EDAC sbridge: Ver: 1.1.2 [ 21.217786] Key type dns_resolver registered [ 21.592727] NFS: Registering the id_resolver key type [ 21.596537] Key type id_resolver registered [ 21.598938] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-client login: [ 70.588854] libcfs: loading out-of-tree module taints kernel. [ 70.673700] Key type ._llcrypt registered [ 70.675504] Key type .llcrypt registered [ 71.381120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.395178] alg: No test for adler32 (adler32-zlib) [ 72.812273] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 73.626509] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 75.431132] Key type lgssc registered [ 77.148352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 199.257388] Lustre: Mounted lustre-client - version 2.17.57_82_g53f9976 [ 204.182393] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.855733] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing check_logdir /tmp/testlogs/ [ 221.557304] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing yml_node [ 224.736379] Lustre: lustre-OST0000-osc-ffff95e2ca5d8000: disconnect after 23s idle [ 226.007826] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 228.251821] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 230.645318] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 232.372606] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 01:18:49 EDT 2026 [ 249.234886] Lustre: Unmounted lustre-client [ 276.566776] Key type lgssc unregistered [ 276.870927] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.886916] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 276.929668] LNet: Removed LNI 192.168.206.55@tcp [ 277.656166] Key type .llcrypt unregistered [ 277.661500] Key type ._llcrypt unregistered [ 304.276388] Key type ._llcrypt registered [ 304.281774] Key type .llcrypt registered [ 304.522864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 304.546788] alg: No test for adler32 (adler32-zlib) [ 305.478420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 305.601294] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 305.664707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 309.968194] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 310.062716] LNet: 9298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.075134] LNetError: 9298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.101053] LNet: Removed LNI 192.168.206.55@tcp [ 310.642871] Key type .llcrypt unregistered [ 310.648887] Key type ._llcrypt unregistered [ 310.974093] hrtimer: interrupt took 3989158 ns [ 316.531308] Key type ._llcrypt registered [ 316.533189] Key type .llcrypt registered [ 318.151670] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:20:15 (1787548815) [ 318.571525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.615675] alg: No test for adler32 (adler32-zlib) [ 319.599929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.751832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.792241] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.818495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.883775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.935254] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 326.399871] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:20:24 (1787548824) [ 326.560143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.588306] LNet: 10302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.615505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.713204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 333.785975] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:20:31 (1787548831) [ 333.958673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.999154] LNet: 10888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.032438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.136915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 340.797438] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:20:38 (1787548838) [ 341.068386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.113834] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.146430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.260495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 348.298511] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:20:45 (1787548845) [ 348.435555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.468631] LNet: 12058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.505099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.601319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 348.879029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 355.769746] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:20:53 (1787548853) [ 355.862461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.927480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.066992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 356.794177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 357.372447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 364.268149] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:21:01 (1787548861) [ 364.402937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.434579] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.443128] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 364.487596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.578403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 371.514741] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:21:08 (1787548868) [ 371.927679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.997944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.094072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 372.180912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.651296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 372.721521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 373.463979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 373.510874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.095895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 374.151367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 374.786360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 374.836431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.088829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 375.143792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 375.620668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 375.673826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.318471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 376.399903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.286686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.363655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.995536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 378.037183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.716635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.778674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.443717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 379.489090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.061909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 380.147282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.045924] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:21:25 (1787548885) [ 389.130920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.167176] LNet: 14512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.173358] LNet: 14512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 389.197460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.285292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 389.315391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 397.506666] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:21:34 (1787548894) [ 397.723756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.780976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.902636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 397.939991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 405.736781] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:21:43 (1787548903) [ 405.902266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.036837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.203089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 406.256698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 414.448790] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:21:52 (1787548912) [ 414.674578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.728626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.824136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.878125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.108266] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:21:58 (1787548918) [ 421.428946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.456425] LNet: 16865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.465124] LNet: 16865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 421.489487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.580957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.622917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.227266] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:22:05 (1787548925) [ 428.386540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.471524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.632513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.688587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 435.767858] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:22:13 (1787548933) [ 435.904392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.015753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.102972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.525763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.443530] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:22:21 (1787548941) [ 443.572752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.618106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.853904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.099546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.052461] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:22:29 (1787548949) [ 452.181591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.229885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.352765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.542922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.886536] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:22:37 (1787548957) [ 460.078460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.156385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.650696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.943625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.928819] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:22:45 (1787548965) [ 468.104750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.204254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.340262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.402948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.036312] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:22:53 (1787548973) [ 476.192256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.251390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.401975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.515839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.596277] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:23:01 (1787548981) [ 483.653068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.680472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.772742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 490.241588] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:23:07 (1787548987) [ 490.431024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.455481] LNet: 22304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.464187] LNet: 22304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 490.504289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.585514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 496.778888] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:23:14 (1787548994) [ 496.951779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.005594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.060479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 497.643431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 504.775692] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:23:21 (1787549001) [ 504.917620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.980138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.101397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 505.136729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 511.722709] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:23:29 (1787549009) [ 511.941070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.033457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.156503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 512.200346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 518.967126] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:23:36 (1787549016) [ 519.057959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.126644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.187609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 519.213282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 526.268699] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:23:43 (1787549023) [ 526.417982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.496138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.623587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 526.704222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 534.442527] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:23:51 (1787549031) [ 534.567357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.627962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.749926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 534.844623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 534.885241] LNetError: 25837:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 534.938203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 534.993765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 541.428142] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:23:58 (1787549038) [ 541.586407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.669788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.819193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OnZjkKykJz [ 541.930272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OnZjkKykJz [ 541.997548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OnZjkKykJz [ 547.966596] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:24:05 (1787549045) [ 548.140637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.232882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.284297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 548.327661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 548.387504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 554.527830] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:24:12 (1787549052) [ 554.713312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.771076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.888590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 554.933606] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 556.276745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 557.805085] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 559.791483] Key type lgssc registered [ 561.215738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 564.656212] LNetError: 27599:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.55@tcp99 has no available nets [ 567.145971] Key type lgssc unregistered [ 567.582855] LNetError: 28710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.607446] LNet: Removed LNI 192.168.206.55@tcp [ 568.129965] Key type .llcrypt unregistered [ 568.132343] Key type ._llcrypt unregistered [ 574.262938] Key type ._llcrypt registered [ 574.264776] Key type .llcrypt registered [ 575.573469] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:24:32 (1787549072) [ 576.113464] Key type .llcrypt unregistered [ 576.115735] Key type ._llcrypt unregistered [ 582.220687] Key type ._llcrypt registered [ 582.223830] Key type .llcrypt registered [ 582.384268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.398929] alg: No test for adler32 (adler32-zlib) [ 583.341174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.425059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 583.445895] LNet: 29630:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 583.454544] LNet: 29630:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 583.466725] LNet: 29630:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 583.484853] LNet: Message forwarding will be enabled in 2s [ 586.592967] LNet: Message forwarding enabled [ 586.961690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 587.036983] LNet: Message forwarding will stop in 2s to 6s [ 590.690632] LNet: Message forwarding disabled [ 591.582539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.605359] LNet: 29728:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 591.614372] LNet: 29728:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 591.627064] LNet: 29728:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 591.650865] LNet: Message forwarding will be enabled in 2s [ 591.874275] LNet: 29741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.570494] Key type .llcrypt unregistered [ 592.575875] Key type ._llcrypt unregistered [ 604.007288] Key type ._llcrypt registered [ 604.009845] Key type .llcrypt registered [ 605.389185] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:25:02 (1787549102) [ 605.964589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 605.994336] alg: No test for adler32 (adler32-zlib) [ 606.962635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.081325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 614.036977] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:25:11 (1787549111) [ 614.149805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.196832] LNet: 31141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.232929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.304610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 614.346840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 614.391532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 614.414416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 614.434179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 614.469482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 614.520653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 614.567266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 614.614971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 614.639942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 614.666537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 614.706032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 614.740626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 614.777631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 614.805909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 614.849680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 614.875934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 614.918304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 614.959683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 614.979403] LNetError: 31191:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 621.228193] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:25:18 (1787549118) [ 621.395649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.437338] LNet: 31760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.492427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.571969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.135347] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:25:27 (1787549127) [ 632.118420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg655-server [ 634.583932] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:25:31 (1787549131) [ 635.047744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.097556] LNet: 32693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.134603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.233518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.269418] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 636.648878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[56-60]@tcp [ 636.684625] LNet: 32826:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 636.772410] LNetError: 32697:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.56@tcp is being used as a gateway but routing feature is not turned on [ 636.794590] LNetError: 32697:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.56@tcp has gone from up to down [ 639.852699] LNetError: 32829:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.60@tcp has gone from up to down [ 639.876045] LNetError: 32829:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 645.024208] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:25:42 (1787549142) [ 646.931385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg655-server [ 648.759828] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:25:46 (1787549146) [ 648.851336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.886193] LNet: 33757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.898024] LNetError: 33757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.916882] LNet: Removed LNI 192.168.206.55@tcp [ 648.947426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.039714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.079608] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 649.825363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[56-61/2]@tcp [ 649.856308] LNet: 33880:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 649.870730] LNet: 33880:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 649.913516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[56-61/2]@tcp [ 649.922536] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.58@tcp is being used as a gateway but routing feature is not turned on [ 649.958927] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 649.980694] LNetError: 33767:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.58@tcp has gone from up to down [ 655.653680] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:25:53 (1787549153) [ 655.839821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.862484] LNet: 34472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.873715] LNetError: 34472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.896817] LNet: Removed LNI 192.168.206.55@tcp [ 655.955583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.052524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 656.098249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.123190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.175647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.222892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.256367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.291815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 656.397568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.466281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.534783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.662376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.690317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.723951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 656.798675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.851725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.924929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.973619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.009763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.081916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 657.180819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.229916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.316614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.397768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.437526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.507685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 657.576125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.606606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.658422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.738942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.792217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.847874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 657.896780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.922195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.978269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.951350] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:26:02 (1787549162) [ 666.685389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg655-server [ 668.513644] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:26:05 (1787549165) [ 670.761317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg655-server [ 672.688790] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:26:09 (1787549169) [ 672.831769] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.839492] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 673.464486] Key type .llcrypt unregistered [ 673.468540] Key type ._llcrypt unregistered [ 679.737047] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 679.892548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 679.900880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 679.913903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 681.609634] Key type ._llcrypt registered [ 681.614767] Key type .llcrypt registered [ 681.848412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.873498] alg: No test for adler32 (adler32-zlib) [ 682.805417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.883520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.911602] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 683.592841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.611651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 685.501451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 685.557246] LNet: Removed LNI 10.1.2.3@tcp [ 685.679594] LNet: 36595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.687544] LNetError: 36595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.713283] LNet: Removed LNI 192.168.206.55@tcp [ 686.368889] Key type .llcrypt unregistered [ 686.372654] Key type ._llcrypt unregistered [ 698.435665] Key type ._llcrypt registered [ 698.443423] Key type .llcrypt registered [ 699.806450] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:26:37 (1787549197) [ 700.648671] Key type .llcrypt unregistered [ 700.650476] Key type ._llcrypt unregistered [ 709.842277] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.048887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.058930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.071207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.748840] Key type ._llcrypt registered [ 712.750439] Key type .llcrypt registered [ 712.981172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.007614] alg: No test for adler32 (adler32-zlib) [ 713.975939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.048657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.071193] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 714.855513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 714.892305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.625900] LNet: 38107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.633115] LNetError: 38107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 717.640383] LNet: Removed LNI 10.1.2.3@tcp [ 717.652587] LNet: Removed LNI 192.168.206.55@tcp [ 718.476889] Key type .llcrypt unregistered [ 718.480410] Key type ._llcrypt unregistered [ 730.632834] Key type ._llcrypt registered [ 730.635453] Key type .llcrypt registered [ 732.242934] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:27:09 (1787549229) [ 732.874287] Key type .llcrypt unregistered [ 732.876978] Key type ._llcrypt unregistered [ 739.297683] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.653851] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.661971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.665304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.233634] Key type ._llcrypt registered [ 742.239549] Key type .llcrypt registered [ 742.506638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.546899] alg: No test for adler32 (adler32-zlib) [ 743.511726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.613738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 743.662773] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 746.076521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 746.113431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.073164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 748.101331] LNet: Removed LNI 10.1.2.3@tcp [ 748.126586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 748.170869] LNet: Removed LNI 10.1.2.31@tcp [ 748.185027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 748.216040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.934907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 748.950183] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 749.921752] LNet: 40035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.927799] LNetError: 40035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 749.937601] LNet: Removed LNI 10.1.2.31@tcp [ 749.953802] LNet: Removed LNI 10.1.2.3@tcp [ 750.647570] Key type .llcrypt unregistered [ 750.649973] Key type ._llcrypt unregistered [ 762.503292] Key type ._llcrypt registered [ 762.505337] Key type .llcrypt registered [ 764.195898] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:27:41 (1787549261) [ 764.680757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.703277] alg: No test for adler32 (adler32-zlib) [ 765.665720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.749405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.782358] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 766.672658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.688323] LNet: 40992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.696501] LNetError: 40992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.720094] LNet: Removed LNI 192.168.206.55@tcp [ 766.751482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.851457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.55@tcp [ 766.882652] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 767.623820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.55@tcp [ 767.673364] LNet: Removed LNI 192.168.206.55@tcp [ 767.788938] LNet: 41127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.491408] Key type .llcrypt unregistered [ 768.492749] Key type ._llcrypt unregistered [ 778.917371] Key type ._llcrypt registered [ 778.919815] Key type .llcrypt registered [ 780.289508] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:27:57 (1787549277) [ 782.422890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg655-server [ 783.999851] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:28:01 (1787549281) [ 784.498321] Key type .llcrypt unregistered [ 784.501866] Key type ._llcrypt unregistered [ 790.713405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.939701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.950666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.963475] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.533189] Key type ._llcrypt registered [ 793.535208] Key type .llcrypt registered [ 793.726887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.747353] alg: No test for adler32 (adler32-zlib) [ 794.703379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.780121] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 795.866761] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 798.695153] LNet: 42966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.703787] LNetError: 42966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.709756] LNet: Removed LNI 192.168.206.55@tcp1 [ 798.722629] LNet: Removed LNI 10.1.2.3@tcp2 [ 799.353529] Key type .llcrypt unregistered [ 799.358278] Key type ._llcrypt unregistered [ 813.111637] Key type ._llcrypt registered [ 813.115525] Key type .llcrypt registered [ 814.618311] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:28:31 (1787549311) [ 814.930282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.946510] alg: No test for adler32 (adler32-zlib) [ 815.993710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.061888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.088465] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 817.048897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.084297] LNet: 43940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.092530] LNetError: 43940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.123689] LNet: Removed LNI 192.168.206.55@tcp [ 817.154242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.238498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 817.284970] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 824.510553] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:28:42 (1787549322) [ 824.785130] LNet: 44530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.801226] LNetError: 44530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.815746] LNet: Removed LNI 192.168.206.55@tcp [ 825.470753] Key type .llcrypt unregistered [ 825.474653] Key type ._llcrypt unregistered [ 832.675352] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.927209] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.933228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.935774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.560859] Key type ._llcrypt registered [ 835.564805] Key type .llcrypt registered [ 835.772913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.795276] alg: No test for adler32 (adler32-zlib) [ 836.810865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.902170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.920089] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 837.717095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 837.741906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.949875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.982950] LNet: 45276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.997913] LNetError: 45276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.011695] LNet: Removed LNI 10.1.2.3@tcp [ 840.041458] LNet: Removed LNI 192.168.206.55@tcp [ 840.088345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.154387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.175888] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 840.186598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.668836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.692448] LNet: 45303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.698226] LNetError: 45303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.702968] LNetError: 45303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.708920] LNet: Removed LNI 10.1.2.3@tcp [ 840.718380] LNet: Removed LNI 192.168.206.55@tcp [ 840.736191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.797862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.837086] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 840.843323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.421632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 841.457641] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 842.402516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.422262] LNet: 45465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.429708] LNetError: 45465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.437372] LNetError: 45465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 842.449395] LNet: Removed LNI 10.1.2.3@tcp [ 842.451245] LNet: Removed LNI 192.168.206.55@tcp [ 842.499350] LNet: Removed LNI 192.168.206.55@tcp2 [ 842.548048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.609461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 842.656702] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 842.662313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.682793] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 850.541351] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:29:08 (1787549348) [ 850.771625] LNet: 46063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.779917] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.789324] LNetError: 46063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 850.796515] LNet: Removed LNI 10.1.2.3@tcp [ 850.799918] LNet: Removed LNI 192.168.206.55@tcp [ 850.814323] LNet: Removed LNI 192.168.206.55@tcp2 [ 851.427639] Key type .llcrypt unregistered [ 851.431100] Key type ._llcrypt unregistered [ 857.397741] Key type ._llcrypt registered [ 857.399439] Key type .llcrypt registered [ 857.609821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.624470] alg: No test for adler32 (adler32-zlib) [ 858.607488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.678473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.714252] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 859.452840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.466291] LNet: 46461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.470619] LNetError: 46461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.485150] LNet: Removed LNI 192.168.206.55@tcp [ 859.501711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.568270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.610403] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 860.038608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.073671] LNet: 46491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.086420] LNetError: 46491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.106949] LNet: Removed LNI 192.168.206.55@tcp [ 860.127841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.208935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.238802] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 860.821905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.870818] LNet: Removed LNI 192.168.206.55@tcp [ 860.884938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.942473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.964851] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 861.303559] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.437828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.442585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.445836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.798658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.823907] LNet: 46600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.838454] LNet: 46600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 863.853712] LNetError: 46600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.859437] LNetError: 46600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 863.872792] LNet: Removed LNI 192.168.206.55@tcp [ 863.904925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.020791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.085789] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 871.215990] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:29:28 (1787549368) [ 871.443654] LNet: 47197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.448490] LNetError: 47197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.462202] LNet: Removed LNI 192.168.206.55@tcp [ 872.119823] Key type .llcrypt unregistered [ 872.122163] Key type ._llcrypt unregistered [ 878.069354] Key type ._llcrypt registered [ 878.072986] Key type .llcrypt registered [ 878.268977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.282619] alg: No test for adler32 (adler32-zlib) [ 879.171684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.211947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.265581] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 880.038559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.053371] LNet: 47599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.059601] LNetError: 47599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.071877] LNet: Removed LNI 192.168.206.55@tcp [ 880.089438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.150381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 880.194204] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 887.917776] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:29:45 (1787549385) [ 888.006036] LNet: 48182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.020744] LNetError: 48182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.051287] LNet: Removed LNI 192.168.206.55@tcp [ 888.719874] Key type .llcrypt unregistered [ 888.725237] Key type ._llcrypt unregistered [ 895.095743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.235884] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.244101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.258222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.808398] Key type ._llcrypt registered [ 897.811431] Key type .llcrypt registered [ 898.014196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.024473] alg: No test for adler32 (adler32-zlib) [ 898.933384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.003639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.043699] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 899.787254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.815603] LNet: 48642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.821842] LNetError: 48642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.838321] LNet: Removed LNI 192.168.206.55@tcp [ 899.855595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.927497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 899.944253] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 906.019029] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:30:03 (1787549403) [ 906.158441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.195093] LNet: 49236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.210973] LNetError: 49236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.233451] LNet: Removed LNI 192.168.206.55@tcp [ 906.293205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.435626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 912.309948] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:30:09 (1787549409) [ 912.416263] LNet: 49818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.119888] Key type .llcrypt unregistered [ 913.122639] Key type ._llcrypt unregistered [ 918.689972] Key type ._llcrypt registered [ 918.696475] Key type .llcrypt registered [ 918.927254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.944738] alg: No test for adler32 (adler32-zlib) [ 919.840440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.907439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.927349] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 920.683614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.703264] LNet: 50245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.710781] LNetError: 50245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.720578] LNet: Removed LNI 192.168.206.55@tcp [ 920.734384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.793357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 920.828316] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 927.420524] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:30:24 (1787549424) [ 927.549917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.572869] LNet: 50835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.578206] LNetError: 50835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.621105] LNet: Removed LNI 192.168.206.55@tcp [ 927.675314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.757640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 932.817796] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:30:30 (1787549430) [ 934.585709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 936.108617] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:30:33 (1787549433) [ 937.855404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 940.105530] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:30:37 (1787549437) [ 941.562745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 943.163629] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:30:40 (1787549440) [ 944.506618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 946.592093] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:30:43 (1787549443) [ 948.334099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 949.916539] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:30:47 (1787549447) [ 950.105522] LNet: 52888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.972953] Key type .llcrypt unregistered [ 950.974344] Key type ._llcrypt unregistered [ 957.705185] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.848920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.859815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.868749] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.651938] Key type ._llcrypt registered [ 960.654356] Key type .llcrypt registered [ 960.940496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.964463] alg: No test for adler32 (adler32-zlib) [ 961.938781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.030798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.049886] LNet: Added LNI 192.168.206.55@tcp [8/512/0/180] [ 962.054698] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 962.061044] LNet: 53214:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.152387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.168894] LNet: 53233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.174357] LNetError: 53233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.184286] LNet: Removed LNI 10.1.2.3@tcp [ 962.186575] LNetError: 53233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.210793] LNet: Removed LNI 192.168.206.55@tcp [ 962.233277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.307819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.358653] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 962.483513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.509919] LNet: 53257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.520326] LNetError: 53257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.555781] LNet: Removed LNI 192.168.206.55@tcp [ 962.630588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.728178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.759103] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 962.770690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.772320] LNet: 53265:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.873477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.897828] LNet: 53283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.902612] LNetError: 53283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.907374] LNet: Removed LNI 10.1.2.3@tcp [ 962.909336] LNetError: 53283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.921682] LNet: Removed LNI 192.168.206.55@tcp [ 962.960961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.067475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.111735] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 963.120950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.129645] LNet: 53291:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.399421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.420909] LNet: 53317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.428603] LNetError: 53317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.435049] LNet: Removed LNI 10.1.2.3@tcp [ 963.438517] LNetError: 53317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.450858] LNet: Removed LNI 192.168.206.55@tcp [ 963.493623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.573611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.593528] LNet: Added LNI 192.168.206.55@tcp [8/256/0/182] [ 963.600466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.704532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.738717] LNet: 53345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.745698] LNetError: 53345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.753656] LNet: Removed LNI 10.1.2.3@tcp [ 963.756481] LNetError: 53345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.767537] LNet: Removed LNI 192.168.206.55@tcp [ 963.785514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.872667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.898299] LNet: Added LNI 192.168.206.55@tcp [8/256/0/182] [ 963.904945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.908314] LNet: 53353:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 964.004658] LNet: Removed LNI 10.1.2.3@tcp [ 964.018076] LNet: Removed LNI 192.168.206.55@tcp [ 964.951870] Key type .llcrypt unregistered [ 964.955807] Key type ._llcrypt unregistered [ 978.453204] Key type ._llcrypt registered [ 978.459332] Key type .llcrypt registered [ 980.148921] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:31:17 (1787549477) [ 982.351809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 984.191689] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:31:21 (1787549481) [ 984.686223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 984.704365] alg: No test for adler32 (adler32-zlib) [ 985.657443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.715756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.736075] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 986.499172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.55 [ 986.521752] LNet: 54623:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 986.529449] LNet: Removed LNI 192.168.206.55@tcp [ 993.201138] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:31:30 (1787549490) [ 993.508506] LNet: 55194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.518803] LNetError: 55194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.539220] LNet: Removed LNI 192.168.206.55@tcp [ 994.119521] Key type .llcrypt unregistered [ 994.129208] Key type ._llcrypt unregistered [ 999.966527] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.222906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.854600] Key type ._llcrypt registered [ 1002.862835] Key type .llcrypt registered [ 1003.001495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.025329] alg: No test for adler32 (adler32-zlib) [ 1004.319829] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.950114] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:31:47 (1787549507) [ 1010.181626] LNet: 56111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.190654] LNetError: 56111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.206538] LNet: Removed LNI 10.1.2.3@tcp [ 1010.814904] Key type .llcrypt unregistered [ 1010.819637] Key type ._llcrypt unregistered [ 1016.392797] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1016.575384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1016.577574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1016.584602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1019.232170] Key type ._llcrypt registered [ 1019.235586] Key type .llcrypt registered [ 1019.373368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.383955] alg: No test for adler32 (adler32-zlib) [ 1020.281577] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.251768] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:32:03 (1787549523) [ 1026.601615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1028.111897] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:32:05 (1787549525) [ 1028.228856] LNet: 57292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.236428] LNetError: 57292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1028.256680] LNet: Removed LNI 10.1.2.3@tcp [ 1028.840592] Key type .llcrypt unregistered [ 1028.842856] Key type ._llcrypt unregistered [ 1034.310877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1034.501896] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1034.505815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1034.512293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.018932] Key type ._llcrypt registered [ 1036.021255] Key type .llcrypt registered [ 1036.207898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.221496] alg: No test for adler32 (adler32-zlib) [ 1037.112462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.191399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.825124] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:32:21 (1787549541) [ 1044.161980] LNet: 58524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.166835] LNetError: 58524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.179691] LNet: Removed LNI 10.1.2.3@tcp [ 1044.799438] Key type .llcrypt unregistered [ 1044.804385] Key type ._llcrypt unregistered [ 1049.796960] Key type ._llcrypt registered [ 1049.799743] Key type .llcrypt registered [ 1049.947242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.958400] alg: No test for adler32 (adler32-zlib) [ 1050.849804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1050.880486] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1058.071301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1058.756726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_interrupt -r 1 [ 1058.789988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_interrupt -r 1 [ 1058.825482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1058.846433] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1059.097713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.103342] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1059.145529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.186285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.221717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.267902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.318698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.348143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.395208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.416326] LNet: 58965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.445545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.488660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.543910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1059.608322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1059.649438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1059.711917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1059.765756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.244956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_dropped -r 1 [ 1060.277316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_dropped -r 1 [ 1060.300862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1060.316279] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1060.319752] LNet: Skipped 1 previous similar message [ 1060.508718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.527967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1060.553705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.577776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.607305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.635676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.672334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.699346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.715191] LNet: 59074:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.736239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.765259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.797349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.846473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.878893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.909415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.943411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1061.429757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_aborted -r 1 [ 1061.458668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_aborted -r 1 [ 1061.483765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1061.502071] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1061.672964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.707407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.745945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.773259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.823980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.849441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.874129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.902612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.916134] LNet: 59184:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.934845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.964173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.003626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.041355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.076563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.107931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.137379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.610117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_no_route -r 1 [ 1062.644213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_no_route -r 1 [ 1062.691440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1062.944028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.977549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.026211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.069287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.126556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.161601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.194298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.238239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.285126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.320742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.358806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.388332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.414588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.444179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.474890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1063.965882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_timeout -r 1 [ 1064.017147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_timeout -r 1 [ 1064.045478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1064.065768] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1064.069973] LNet: Skipped 1 previous similar message [ 1064.302676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.326180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.349161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.375859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.399443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.426273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.461468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.509245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.535869] LNet: 59402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.545888] LNet: 59402:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1064.575707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.641323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.680606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.710917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.736992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.770790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.800851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.305278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_error -r 1 [ 1065.344824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_error -r 1 [ 1065.396230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1065.611325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.653656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.693930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.727580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.768250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.806859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.836188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.868413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.900382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.925455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.953253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.981347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.005215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.037261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.065698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.279503] LNet: 59556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.286248] LNetError: 59556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.296794] LNet: Removed LNI 192.168.206.55@tcp [ 1066.672489] Key type .llcrypt unregistered [ 1066.674119] Key type ._llcrypt unregistered [ 1083.195689] Key type ._llcrypt registered [ 1083.197447] Key type .llcrypt registered [ 1084.383743] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:33:01 (1787549581) [ 1085.127784] Key type .llcrypt unregistered [ 1085.130815] Key type ._llcrypt unregistered [ 1091.850449] Key type ._llcrypt registered [ 1091.853482] Key type .llcrypt registered [ 1092.191693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.235136] alg: No test for adler32 (adler32-zlib) [ 1093.212380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1093.246667] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1101.610681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1114.120937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.192142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1114.233407] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1115.561984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_interrupt -r 1 [ 1115.609145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_interrupt -r 1 [ 1115.660070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_interrupt -r 1 [ 1115.707547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_interrupt -r 1 [ 1115.774123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_interrupt -r 1 [ 1115.828093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_interrupt -r 1 [ 1115.865425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_interrupt -r 1 [ 1115.907591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_interrupt -r 1 [ 1115.935425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1115.956811] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1115.963801] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1116.296563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.361114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.396797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.432414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.467312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.498616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.526800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.551269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.572420] LNet: 61028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.603029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.636407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.674722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.704623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.730356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.758725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.786342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.372431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_dropped -r 1 [ 1117.424660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_dropped -r 1 [ 1117.474657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_dropped -r 1 [ 1117.520082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_dropped -r 1 [ 1117.566914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_dropped -r 1 [ 1117.594302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_dropped -r 1 [ 1117.616804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_dropped -r 1 [ 1117.673381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_dropped -r 1 [ 1117.736819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1117.773164] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1117.783492] LNet: Skipped 2 previous similar messages [ 1118.133995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.189807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.242479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.285995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.344940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.391764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.458448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.523861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.562161] LNet: 61150:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.605901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.650875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.684638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.728380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.792990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.820384] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1118.832017] LNet: Skipped 3 previous similar messages [ 1118.866513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.895993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.419807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_aborted -r 1 [ 1119.470926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_aborted -r 1 [ 1119.519556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_aborted -r 1 [ 1119.566562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_aborted -r 1 [ 1119.603326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_aborted -r 1 [ 1119.639660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_aborted -r 1 [ 1119.666342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_aborted -r 1 [ 1119.706452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_aborted -r 1 [ 1119.762882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1119.991451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.016960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.055337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.094707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.130959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.190308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.259555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.286287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.304844] LNet: 61271:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.335179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.383693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.414279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.474624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.536936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.623697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.679531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.287581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_no_route -r 1 [ 1121.320936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_no_route -r 1 [ 1121.351625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_no_route -r 1 [ 1121.385389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_no_route -r 1 [ 1121.422659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_no_route -r 1 [ 1121.460388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_no_route -r 1 [ 1121.491374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_no_route -r 1 [ 1121.525634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_no_route -r 1 [ 1121.562607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1121.583569] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1121.587392] LNet: Skipped 3 previous similar messages [ 1121.800965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.834785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.865959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.910427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.954304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.974275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.010124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.034775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.061320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.098844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.147462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.185596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.230141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.284558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.333487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.134668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_timeout -r 1 [ 1123.177153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_timeout -r 1 [ 1123.214667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_timeout -r 1 [ 1123.286937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_timeout -r 1 [ 1123.346535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_timeout -r 1 [ 1123.411584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_timeout -r 1 [ 1123.499707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_timeout -r 1 [ 1123.561653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_timeout -r 1 [ 1123.627951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1123.948455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.995767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.047766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.145725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.219917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.276754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.368449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.413717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.438188] LNet: 61514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.444520] LNet: 61514:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1124.475677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.535247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.575100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.619726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.674532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.702957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.750077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.313604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_error -r 1 [ 1125.354471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_error -r 1 [ 1125.384966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_error -r 1 [ 1125.429577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_error -r 1 [ 1125.489326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_error -r 1 [ 1125.581835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_error -r 1 [ 1125.653877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_error -r 1 [ 1125.698706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_error -r 1 [ 1125.770246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1125.788752] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1125.792592] LNet: Skipped 5 previous similar messages [ 1126.117117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.160605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.198758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.275103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.349053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.427570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.500620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.546411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.578514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.615515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.647577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.707913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.754734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.822112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.869404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.420912] LNet: 61716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.440242] LNetError: 61716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.449606] LNet: Removed LNI 192.168.206.55@tcp [ 1133.493122] LNet: Removed LNI 192.168.206.55@tcp1 [ 1133.886695] Key type .llcrypt unregistered [ 1133.892569] Key type ._llcrypt unregistered [ 1150.858863] Key type ._llcrypt registered [ 1150.863446] Key type .llcrypt registered [ 1152.267280] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:34:09 (1787549649) [ 1152.963337] Key type .llcrypt unregistered [ 1152.965432] Key type ._llcrypt unregistered [ 1159.330426] Key type ._llcrypt registered [ 1159.332375] Key type .llcrypt registered [ 1159.528768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.559281] alg: No test for adler32 (adler32-zlib) [ 1160.506207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1160.537375] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1167.622094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1168.188974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_dropped -r 1 [ 1168.243663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_dropped -r 1 [ 1168.275390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1168.295431] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1168.550826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.590709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.619974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.648151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.683051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.726859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.770366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.826385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.846520] LNet: 62970:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.866031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.898603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.940632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.986191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.033025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.079433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.110799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.649905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_error -r 1 [ 1169.689923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_error -r 1 [ 1169.716549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1169.738760] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1169.989639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.012274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.036739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.057715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.081375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.128885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.159132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.186698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.208225] LNet: 63079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.227587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.268641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.303412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.358723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.400063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.438467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.479954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.892696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_timeout -r 1 [ 1170.944631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_timeout -r 1 [ 1170.993871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1171.022645] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1171.357721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.393393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.429715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.468472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.501236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.539988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.594387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.647183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.664917] LNet: 63188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.685949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.719289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.759384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.784553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.822811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.855757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.889915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.216738] LNet: 63233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.221772] LNetError: 63233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.234726] LNet: Removed LNI 192.168.206.55@tcp [ 1172.620180] Key type .llcrypt unregistered [ 1172.625076] Key type ._llcrypt unregistered [ 1188.799845] Key type ._llcrypt registered [ 1188.802928] Key type .llcrypt registered [ 1189.994958] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:34:47 (1787549687) [ 1190.831252] Key type .llcrypt unregistered [ 1190.835145] Key type ._llcrypt unregistered [ 1197.259247] Key type ._llcrypt registered [ 1197.262926] Key type .llcrypt registered [ 1197.503758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1197.518931] alg: No test for adler32 (adler32-zlib) [ 1198.469796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1198.514446] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1206.610726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1214.320086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1214.372073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1214.395866] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1215.875569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_dropped -r 1 [ 1215.930881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e remote_dropped -r 1 [ 1215.974407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_dropped -r 1 [ 1216.025746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e remote_dropped -r 1 [ 1216.078092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e remote_dropped -r 1 [ 1216.138458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e remote_dropped -r 1 [ 1216.203469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e remote_dropped -r 1 [ 1216.253831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e remote_dropped -r 1 [ 1216.285929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1216.317343] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1216.324018] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.155@tcp [ 1216.552960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.591407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.622278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.660244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.699416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.732694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.797054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.847719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.880649] LNet: 64705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.912650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.957932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.003690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.053899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.094617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.133382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.167082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1217.195762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.838609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_error -r 1 [ 1217.867269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e remote_error -r 1 [ 1217.895245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_error -r 1 [ 1217.924142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e remote_error -r 1 [ 1217.959796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e remote_error -r 1 [ 1218.015169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e remote_error -r 1 [ 1218.047844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e remote_error -r 1 [ 1218.077389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e remote_error -r 1 [ 1218.112875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1218.133132] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1218.141543] LNet: Skipped 2 previous similar messages [ 1218.344663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.383152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.430066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.461085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.491526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.529622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.566385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.597304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.614616] LNet: 64828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.635470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.673587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.704906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.742600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.791847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.828914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.860673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.921721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.444379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_timeout -r 1 [ 1219.506045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e remote_timeout -r 1 [ 1219.564135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_timeout -r 1 [ 1219.646922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e remote_timeout -r 1 [ 1219.752500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e remote_timeout -r 1 [ 1219.810394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e remote_timeout -r 1 [ 1219.857733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e remote_timeout -r 1 [ 1219.904178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e remote_timeout -r 1 [ 1219.946668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.155@tcp [ 1219.977437] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1220.281537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.330710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.378673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.423515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.472961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.500672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.531828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.556651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.592587] LNet: 64951:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.622984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.673184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.740587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.787421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1220.848944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.919489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.969684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1221.027482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.650593] LNet: 65033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.655659] LNetError: 65033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.663801] LNet: Removed LNI 192.168.206.55@tcp [ 1226.677951] LNet: Removed LNI 192.168.206.55@tcp1 [ 1227.185552] Key type .llcrypt unregistered [ 1227.188501] Key type ._llcrypt unregistered [ 1245.895685] Key type ._llcrypt registered [ 1245.897798] Key type .llcrypt registered [ 1247.434382] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:35:44 (1787549744) [ 1248.695440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1250.719734] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:35:48 (1787549748) [ 1251.644403] Key type .llcrypt unregistered [ 1251.647541] Key type ._llcrypt unregistered [ 1257.826655] Key type ._llcrypt registered [ 1257.835480] Key type .llcrypt registered [ 1258.117663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.173268] alg: No test for adler32 (adler32-zlib) [ 1259.158640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.184407] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1266.535682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1267.410652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e network_timeout -r 1 [ 1267.449647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e network_timeout -r 1 [ 1267.477702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1267.493415] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1267.731228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.753547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.781109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.804765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.832721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.865585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.887315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.910582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.922415] LNet: 66580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1267.940396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.962750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.992535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.026281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.074201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.111990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.374117] LNet: 66623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.385121] LNetError: 66623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.399191] LNet: Removed LNI 192.168.206.55@tcp [ 1268.758666] Key type .llcrypt unregistered [ 1268.761669] Key type ._llcrypt unregistered [ 1285.056405] Key type ._llcrypt registered [ 1285.060510] Key type .llcrypt registered [ 1285.290751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.319992] alg: No test for adler32 (adler32-zlib) [ 1286.257260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.290992] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1292.922090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1300.773160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.802947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1300.820599] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1302.085772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e network_timeout -r 1 [ 1302.117244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e network_timeout -r 1 [ 1302.153477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e network_timeout -r 1 [ 1302.186145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e network_timeout -r 1 [ 1302.227272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e network_timeout -r 1 [ 1302.270870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e network_timeout -r 1 [ 1302.319602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e network_timeout -r 1 [ 1302.360530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e network_timeout -r 1 [ 1302.394556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1302.410711] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1302.605561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.632326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.667734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.690068] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1302.693469] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.155@tcp [ 1302.697630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.732291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.768041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.790791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.814166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.828656] LNet: 67505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.842621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.870497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.893203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.921415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.945429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.977113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.432545] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1307.745477] LNet: 67583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.750406] LNetError: 67583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.761733] LNet: Removed LNI 192.168.206.55@tcp [ 1307.780880] LNet: Removed LNI 192.168.206.55@tcp1 [ 1308.114715] Key type .llcrypt unregistered [ 1308.119732] Key type ._llcrypt unregistered [ 1322.018225] Key type ._llcrypt registered [ 1322.023446] Key type .llcrypt registered [ 1323.009041] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:37:00 (1787549820) [ 1323.397288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1323.413214] alg: No test for adler32 (adler32-zlib) [ 1324.304480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.342784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.359691] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1325.045368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.068579] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1325.950921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 1326.028595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1326.071245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1326.095859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1326.129457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1326.165997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1326.230686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.55@tcp [ 1326.262539] LNetError: 68682:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.55@tcp to recovery [ 1326.367294] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1326.367406] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1328.415810] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1328.636848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1332.511478] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1333.259241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1333.297950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1333.350392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1333.375480] LNet: 68789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.384360] LNetError: 68789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.392910] LNet: Removed LNI 192.168.206.55@tcp [ 1333.404931] LNet: Removed LNI 192.168.206.55@tcp1 [ 1333.424812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.472091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.501467] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1334.152233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1334.173315] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1334.981934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 1335.006789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.050651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1335.079963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1335.113736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1335.151665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1335.179722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1335.207495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.55@tcp [ 1335.223357] LNetError: 69061:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.55@tcp to recovery [ 1335.520087] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1338.653621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1341.664987] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1341.679546] LNet: Skipped 1 previous similar message [ 1346.525958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1346.555886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1346.593389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.230188] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:37:31 (1787549851) [ 1353.430643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.446258] LNet: 69779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.450435] LNetError: 69779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.454877] LNetError: 69779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1353.460325] LNet: Removed LNI 192.168.206.55@tcp [ 1353.471275] LNet: Removed LNI 192.168.206.55@tcp1 [ 1353.504823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.578759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1353.611274] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1354.138152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.160226] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1354.834531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 1354.902116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1354.932199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1354.971196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1355.004119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1355.051934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1355.099047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.55@tcp --health 0 [ 1355.212554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1355.615152] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.55@tcp [ 1356.639686] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1356.646694] LNet: Skipped 1 previous similar message [ 1371.840957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.110330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1372.165750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.204627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.55@tcp --health 500 [ 1372.353652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.122718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1379.223886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.252411] LNet: 70279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.264026] LNetError: 70279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.271599] LNetError: 70279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1379.278106] LNet: Removed LNI 192.168.206.55@tcp [ 1379.294807] LNet: Removed LNI 192.168.206.55@tcp1 [ 1379.308276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.352318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.369351] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1379.894565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.913547] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1380.600898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 1380.631326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.695964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1380.728942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1380.778189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1380.811708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1380.835732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1380.868757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.55@tcp --health 0 [ 1381.407929] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1381.414711] LNet: Skipped 3 previous similar messages [ 1382.152952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.275587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.338703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1392.373376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1400.570502] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:38:17 (1787549897) [ 1401.256881] LNet: 71325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.261676] LNetError: 71325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.265390] LNetError: 71325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1402.271152] LNet: Removed LNI 192.168.206.55@tcp [ 1402.286431] LNet: Removed LNI 192.168.206.55@tcp1 [ 1403.188588] Key type .llcrypt unregistered [ 1403.191550] Key type ._llcrypt unregistered [ 1411.124454] Key type ._llcrypt registered [ 1411.130050] Key type .llcrypt registered [ 1411.494077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1411.531607] alg: No test for adler32 (adler32-zlib) [ 1412.523290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1412.569562] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1421.025497] Lustre: DEBUG MARKER: Initial discovery [ 1421.242647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.155@tcp [ 1423.257086] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1423.393509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1423.441328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.155@tcp [ 1423.472879] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1423.506047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1423.556760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.155@tcp -r 1 -m GET -l 3 [ 1423.593366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.155@tcp -r 1 -m GET -e local_error [ 1423.622531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.155@tcp -r 1 -m PUT -l 6 [ 1423.847183] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1424.836368] Lustre: DEBUG MARKER: Force oleg655-client.virtnet to discover 192.168.206.155@tcp (in background) [ 1424.903678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.155@tcp [ 1426.308262] Lustre: DEBUG MARKER: Force oleg655-server to discover 192.168.206.55@tcp [ 1428.001283] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1428.506346] Lustre: DEBUG MARKER: Wait for 72086 [ 1430.089501] Lustre: DEBUG MARKER: Finished wait on 72086 [ 1430.234520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1430.281255] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1430.286824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.601817] LNet: 72547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.612929] LNetError: 72547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.634493] LNet: Removed LNI 192.168.206.55@tcp [ 1431.068679] Key type .llcrypt unregistered [ 1431.073837] Key type ._llcrypt unregistered [ 1446.475279] Key type ._llcrypt registered [ 1446.479078] Key type .llcrypt registered [ 1447.894334] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:39:05 (1787549945) [ 1448.488957] Key type .llcrypt unregistered [ 1448.492905] Key type ._llcrypt unregistered [ 1453.892497] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1453.957117] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1453.959575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1453.963429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1456.542172] Key type ._llcrypt registered [ 1456.545257] Key type .llcrypt registered [ 1456.779426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1456.797577] alg: No test for adler32 (adler32-zlib) [ 1457.712124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.768230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.802399] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1458.277352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1458.292385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1465.628846] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:39:23 (1787549963) [ 1465.993304] LNet: 74627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.004619] LNetError: 74627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1466.009560] LNet: Removed LNI 10.1.2.3@tcp [ 1466.026495] LNet: Removed LNI 192.168.206.55@tcp [ 1466.641492] Key type .llcrypt unregistered [ 1466.649674] Key type ._llcrypt unregistered [ 1472.844213] Key type ._llcrypt registered [ 1472.852852] Key type .llcrypt registered [ 1473.075239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1473.090383] alg: No test for adler32 (adler32-zlib) [ 1474.009351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.067573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.087715] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1474.770396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1474.787717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1475.836534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.55@tcp [ 1476.895507] LNetError: Unexpected error -22 connecting to 192.168.206.55@tcp at host 192.168.206.55:988 [ 1482.145722] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:39:39 (1787549979) [ 1482.311445] LNet: 75763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.318107] LNetError: 75763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1482.324257] LNet: Removed LNI 10.1.2.3@tcp [ 1482.340460] LNet: Removed LNI 192.168.206.55@tcp [ 1482.947507] Key type .llcrypt unregistered [ 1482.952130] Key type ._llcrypt unregistered [ 1489.244329] Key type ._llcrypt registered [ 1489.250455] Key type .llcrypt registered [ 1489.480108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1489.494711] alg: No test for adler32 (adler32-zlib) [ 1490.450688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.509247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1490.527748] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1491.346949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1491.378829] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 1492.347378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.55@tcp1 --nid 192.168.206.55@tcp2 [ 1496.670612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp1 192.168.206.55@tcp2 [ 1496.694296] LNetError: 76602:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.55@tcp1 to 192.168.206.55@tcp2 - Route table may be misconfigured [ 1496.726243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp2 192.168.206.55@tcp1 [ 1502.514684] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:40:00 (1787550000) [ 1502.651476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.681435] LNet: 77172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.694803] LNetError: 77172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.704433] LNet: Removed LNI 192.168.206.55@tcp1 [ 1503.783184] LNet: Removed LNI 192.168.206.55@tcp2 [ 1503.816413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.869886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1503.914738] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1504.590704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1504.605296] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1505.399169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 1505.440301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.55@tcp [ 1505.489741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp1 192.168.206.55@tcp1 [ 1505.540557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.55@tcp -d 192.168.206.55@tcp -e network_timeout [ 1505.567275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -e network_timeout [ 1505.603498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -e network_timeout [ 1505.664945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -e network_timeout [ 1505.743690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1505.773494] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1505.800336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1505.890315] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1505.924865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.049866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1511.245125] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:40:09 (1787550009) [ 1511.446322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.483287] LNet: 78036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.496527] LNetError: 78036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.501892] LNetError: 78036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.511862] LNet: Removed LNI 192.168.206.55@tcp [ 1512.553703] LNet: Removed LNI 192.168.206.55@tcp1 [ 1512.581626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.639413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1512.880969] LNet: 78077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.288570] Key type .llcrypt unregistered [ 1513.299271] Key type ._llcrypt unregistered [ 1524.565825] Key type ._llcrypt registered [ 1524.568185] Key type .llcrypt registered [ 1525.767829] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:40:23 (1787550023) [ 1526.121228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.141863] alg: No test for adler32 (adler32-zlib) [ 1527.058211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.242772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1529.830820] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:40:27 (1787550027) [ 1529.945739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.977554] LNet: 79173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.040582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.100680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.123668] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1530.887870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1530.921787] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1531.715651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1531.757916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp1 [ 1531.791403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp1 [ 1538.315880] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:40:35 (1787550035) [ 1540.182845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg655-server [ 1541.912700] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:40:39 (1787550039) [ 1543.867514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg655-server [ 1545.929309] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:40:43 (1787550043) [ 1548.161506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg655-server [ 1549.936555] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:40:47 (1787550047) [ 1552.426674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg655-server [ 1554.424415] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:40:51 (1787550051) [ 1557.003724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg655-server [ 1559.475708] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:40:56 (1787550056) [ 1561.055603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg655-server [ 1562.608511] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:41:00 (1787550060) [ 1564.337663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg655-server [ 1565.634088] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:41:03 (1787550063) [ 1567.254325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg655-server [ 1568.902788] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:41:06 (1787550066) [ 1570.611959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg655-server [ 1572.026859] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:41:09 (1787550069) [ 1572.264565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.294318] LNet: 83053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.314809] LNetError: 83053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.322505] LNet: Removed LNI 192.168.206.55@tcp [ 1573.349312] LNet: Removed LNI 192.168.206.55@tcp1 [ 1573.420898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.482608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.522900] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1574.309853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1574.397836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1574.591159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.607875] LNet: 83200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.618590] LNetError: 83200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.625371] LNetError: 83200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1574.646874] LNet: Removed LNI 192.168.206.55@tcp [ 1574.694781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.755864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1574.794345] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1575.383893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1575.518501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.550742] LNet: Removed LNI 192.168.206.55@tcp [ 1576.571777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.648323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.689687] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1577.427078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1577.567782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1579.030486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.060373] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.069504] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1579.083467] LNetError: 83502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.093345] LNetError: 83502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1579.110863] LNet: Removed LNI 192.168.206.55@tcp [ 1579.148441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.220703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1579.261449] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1579.977944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1580.158613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.221525] LNet: Removed LNI 192.168.206.55@tcp [ 1581.252502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.341209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.374470] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1582.165916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1582.226886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1583.620494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.657403] LNet: 83804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.666814] LNet: 83804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1583.673795] LNetError: 83804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.680846] LNetError: 83804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1583.693679] LNet: Removed LNI 192.168.206.55@tcp [ 1583.714369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.776835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1583.810870] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1584.581656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1584.895824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.043733] LNet: Removed LNI 192.168.206.55@tcp [ 1586.112143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.190507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.211440] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1587.243887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1587.359786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1587.591970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.658850] LNet: Removed LNI 192.168.206.55@tcp [ 1588.691651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.756741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1588.783223] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1589.445098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1589.689664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.710637] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.714390] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1590.765405] LNet: Removed LNI 192.168.206.55@tcp [ 1590.802799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.899202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.933075] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1591.590747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1591.651664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1593.131492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.164819] LNet: 84398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.169421] LNet: 84398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1594.229990] LNet: Removed LNI 192.168.206.55@tcp [ 1594.275959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.367286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1594.398661] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1595.274309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1595.572702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.647989] LNet: Removed LNI 192.168.206.55@tcp [ 1596.684205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.774605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.815172] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1597.608927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1597.694401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1598.040962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.064705] LNetError: 84691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.073290] LNetError: 84691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1598.082756] LNet: Removed LNI 192.168.206.55@tcp [ 1598.099777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.143193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1598.171941] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1598.927384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1600.308191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.384347] LNet: Removed LNI 192.168.206.55@tcp [ 1601.416348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.478361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.510262] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1602.217280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1602.318755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.352811] LNet: Removed LNI 192.168.206.55@tcp [ 1602.374985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.461798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1602.495568] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1603.325525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1603.442613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.460768] LNet: Removed LNI 192.168.206.55@tcp [ 1603.475685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.516107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1603.599927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.643732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.701579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1603.792378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.830692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.870740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1603.944637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.972762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.013531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1604.030371] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1604.685671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.705830] LNet: Removed LNI 192.168.206.55@tcp [ 1604.726615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.777608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1610.925269] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:41:48 (1787550108) [ 1611.050922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.082502] LNet: 85861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.099371] LNet: 85861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1611.144636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.205322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1611.225973] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/180] [ 1612.638174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.657791] LNet: Removed LNI 192.168.206.55@tcp231 [ 1612.673385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.727428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1612.747808] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/180] [ 1612.751934] LNet: 86013:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1613.240670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1613.286231] LNet: Removed LNI 192.168.206.55@tcp231 [ 1613.314667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1613.365665] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/180] [ 1614.302420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.318571] LNetError: 86158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.323188] LNetError: 86158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1614.333610] LNet: Removed LNI 192.168.206.55@tcp231 [ 1614.350994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.436933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1614.471694] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/0] [ 1614.476090] LNet: 86166:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.810457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1614.833366] LNet: Removed LNI 192.168.206.55@tcp231 [ 1614.846899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1614.872838] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/0] [ 1615.818668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.871416] LNet: Removed LNI 192.168.206.55@tcp231 [ 1615.905313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.985428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1616.017806] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/60] [ 1616.021752] LNet: 86317:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1616.460873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1616.498465] LNet: Removed LNI 192.168.206.55@tcp231 [ 1616.528260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1616.556460] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/60] [ 1617.752659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.787409] LNet: Removed LNI 192.168.206.55@tcp231 [ 1617.838586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.941345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1617.965685] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/180] [ 1618.417697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1618.450725] LNet: Removed LNI 192.168.206.55@tcp231 [ 1618.467400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1618.490240] LNet: Added LNI 192.168.206.55@tcp231 [8/256/0/180] [ 1625.241403] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:42:02 (1787550122) [ 1625.337590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.411828] LNet: Removed LNI 192.168.206.55@tcp231 [ 1625.442809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.602167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1625.654037] LNet: Added LNI 192.168.206.55@tcp232 [8/256/0/180] [ 1626.393448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1626.430962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1626.462337] LNet: Removed LNI 192.168.206.55@tcp232 [ 1626.476302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1626.507104] LNet: Added LNI 192.168.206.55@tcp232 [8/256/0/180] [ 1627.145487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1627.185359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1627.208752] LNet: Removed LNI 192.168.206.55@tcp232 [ 1627.220556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1627.237880] LNet: Added LNI 192.168.206.55@tcp232 [8/256/0/180] [ 1627.796790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1627.827662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1627.853379] LNet: Removed LNI 192.168.206.55@tcp232 [ 1627.875066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1627.903254] LNet: Added LNI 192.168.206.55@tcp232 [8/256/0/180] [ 1628.450898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1628.493564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1628.526177] LNet: Removed LNI 192.168.206.55@tcp232 [ 1628.545774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1634.374509] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:42:11 (1787550131) [ 1635.315314] Key type .llcrypt unregistered [ 1635.323166] Key type ._llcrypt unregistered [ 1640.522382] Key type ._llcrypt registered [ 1640.524375] Key type .llcrypt registered [ 1640.671455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.687276] alg: No test for adler32 (adler32-zlib) [ 1641.637209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1641.681564] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1649.435888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1658.775044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.855478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1658.899257] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1660.707716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_interrupt -r 2 [ 1660.763733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_interrupt -r 2 [ 1660.793866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_interrupt -r 2 [ 1660.842506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_interrupt -r 2 [ 1660.896542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_interrupt -r 2 [ 1660.945362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_interrupt -r 2 [ 1660.995552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_interrupt -r 2 [ 1661.113795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_interrupt -r 2 [ 1661.198165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1661.259721] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1661.273307] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1661.563347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.612289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1661.645273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.684389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1661.730695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1661.793401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1661.857784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.921481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1661.941792] LNet: 88972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.976298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1662.018459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1662.073705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1662.111705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.181581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1662.224642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1662.881874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_dropped -r 2 [ 1662.924181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_dropped -r 2 [ 1662.991794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_dropped -r 2 [ 1663.052343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_dropped -r 2 [ 1663.115198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_dropped -r 2 [ 1663.175748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_dropped -r 2 [ 1663.226917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_dropped -r 2 [ 1663.282374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_dropped -r 2 [ 1663.320178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1663.345930] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1663.547348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1663.594365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1663.659563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1663.694436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1663.741099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1663.782438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1663.810423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1663.838575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1663.855282] LNet: 89092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.874466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1663.906183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1663.935879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1663.961364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1663.989121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1664.012299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1664.486864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_aborted -r 2 [ 1664.526474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_aborted -r 2 [ 1664.563600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_aborted -r 2 [ 1664.611222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_aborted -r 2 [ 1664.675819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_aborted -r 2 [ 1664.711445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_aborted -r 2 [ 1664.758342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_aborted -r 2 [ 1664.780755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_aborted -r 2 [ 1664.823138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1664.841308] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1665.117176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1665.143992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1665.190424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1665.214910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1665.263183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1665.315226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1665.354860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1665.401706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1665.426072] LNet: 89213:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.454419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1665.500717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1665.551107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1665.607663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1665.635510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1665.663092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1666.124775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_no_route -r 2 [ 1666.204454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_no_route -r 2 [ 1666.280482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_no_route -r 2 [ 1666.331328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_no_route -r 2 [ 1666.372827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_no_route -r 2 [ 1666.431160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_no_route -r 2 [ 1666.504858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_no_route -r 2 [ 1666.573429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_no_route -r 2 [ 1666.640572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1666.968764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1667.022491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1667.073272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1667.148346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1667.213894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1667.303465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1667.351030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1667.399191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1667.441922] LNet: 89333:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1667.482832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1667.530190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1667.585011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1667.639667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1667.720353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1667.757819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1668.342422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e local_timeout -r 2 [ 1668.395756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e local_timeout -r 2 [ 1668.432258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e local_timeout -r 2 [ 1668.468411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e local_timeout -r 2 [ 1668.510614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e local_timeout -r 2 [ 1668.544734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e local_timeout -r 2 [ 1668.572991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e local_timeout -r 2 [ 1668.602057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e local_timeout -r 2 [ 1668.623158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1668.646843] LNet: There was an unexpected network error while writing to 192.168.206.155: rc = -22 [ 1668.652640] LNet: Skipped 1 previous similar message [ 1668.855129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1668.887207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1668.926887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1668.960680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.995198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1669.027193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1669.067990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1669.120196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1669.168725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1669.206715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1669.270660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1669.304607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1669.347193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1669.379222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1670.148832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp -m GET -e remote_dropped -r 2 [ 1670.216845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp -d 192.168.206.155@tcp1 -m GET -e remote_dropped -r 2 [ 1670.287707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp -m GET -e remote_dropped -r 2 [ 1670.345397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp -d 192.168.206.55@tcp1 -m GET -e remote_dropped -r 2 [ 1670.396968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp -m GET -e remote_dropped -r 2 [ 1670.426574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.155@tcp1 -m GET -e remote_dropped -r 2 [ 1670.479946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp -m GET -e remote_dropped -r 2 [ 1670.525562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.55@tcp1 -d 192.168.206.55@tcp1 -m GET -e remote_dropped -r 2 [ 1670.585585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.55@tcp 192.168.206.155@tcp [ 1670.638967] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.155@tcp [ 1670.891202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.929405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.966494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.994082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1671.035877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1671.071932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1671.119782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1671.151675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1671.180116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1671.222098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1671.255863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1671.304916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1671.336747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1671.380392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1676.351946] LNet: 89652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.366158] LNetError: 89652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.373363] LNet: Removed LNI 192.168.206.55@tcp [ 1676.385899] LNet: Removed LNI 192.168.206.55@tcp1 [ 1676.919710] Key type .llcrypt unregistered [ 1676.922870] Key type ._llcrypt unregistered [ 1692.325492] Key type ._llcrypt registered [ 1692.327541] Key type .llcrypt registered [ 1693.759954] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:43:11 (1787550191) [ 1694.292986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1694.307918] alg: No test for adler32 (adler32-zlib) [ 1695.285248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.361627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.382754] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1696.134490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1696.163518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.55@tcp [ 1696.182436] LNetError: 90631:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.55@tcp to recovery [ 1696.254842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.55@tcp [ 1696.350605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1702.242959] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:43:19 (1787550199) [ 1702.466980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.485435] LNet: 91223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.491496] LNetError: 91223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.502688] LNet: Removed LNI 192.168.206.55@tcp [ 1702.521653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.573324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.599289] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1703.424859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1703.461377] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 1704.302819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.55@tcp [ 1704.325259] LNetError: 91489:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.55@tcp to recovery [ 1704.607380] LNet: 1 local NIs in recovery (showing 1): 192.168.206.55@tcp [ 1704.608278] LNet: There was an unexpected network error while writing to 192.168.206.55: rc = -22 [ 1705.602232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.55@tcp [ 1713.048104] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:43:30 (1787550210) [ 1714.587926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg655-server [ 1716.591874] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:43:33 (1787550213) [ 1718.649223] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg655-server [ 1720.231923] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:43:37 (1787550217) [ 1722.366894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg655-server [ 1724.220949] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:43:41 (1787550221) [ 1724.335018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.393779] LNet: 93104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.411206] LNetError: 93104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.428383] LNet: Removed LNI 192.168.206.55@tcp [ 1724.451230] LNet: Removed LNI 192.168.206.55@tcp1 [ 1724.487053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.592529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.619839] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1725.510582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1725.574807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1725.630122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1725.746596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1725.820992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1725.908687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1732.630826] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:43:50 (1787550230) [ 1732.718170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.756368] LNet: 93836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.762766] LNetError: 93836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.774176] LNetError: 93836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1732.792084] LNet: Removed LNI 192.168.206.55@tcp [ 1732.819156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.886105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.913726] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1738.867679] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:43:56 (1787550236) [ 1738.978927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.005748] LNet: 94542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.011639] LNetError: 94542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.026333] LNet: Removed LNI 192.168.206.55@tcp [ 1739.058930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.094912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.110871] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1745.708990] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:44:03 (1787550243) [ 1747.195179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1748.993809] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:44:06 (1787550246) [ 1749.682326] LNet: 95581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.692596] LNetError: 95581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.720313] LNet: Removed LNI 192.168.206.55@tcp [ 1750.524044] Key type .llcrypt unregistered [ 1750.531426] Key type ._llcrypt unregistered [ 1758.071561] Key type ._llcrypt registered [ 1758.079347] Key type .llcrypt registered [ 1758.322934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1758.338813] alg: No test for adler32 (adler32-zlib) [ 1759.403818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1759.450727] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1766.609564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 1772.260751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.155@tcp [ 1773.694363] LNet: 96008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.714805] LNetError: 96008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.748340] LNet: Removed LNI 192.168.206.55@tcp [ 1774.212425] Key type .llcrypt unregistered [ 1774.224197] Key type ._llcrypt unregistered [ 1785.288468] Key type ._llcrypt registered [ 1785.298664] Key type .llcrypt registered [ 1786.576605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1788.069625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1789.741414] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:44:47 (1787550287) [ 1791.284774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1793.003455] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:44:50 (1787550290) [ 1794.722903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg655-server [ 1796.774878] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:44:53 (1787550293) [ 1798.588102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg655-server [ 1800.259302] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:44:57 (1787550297) [ 1801.619887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.681321] alg: No test for adler32 (adler32-zlib) [ 1802.818339] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1803.010292] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1804.711198] Key type lgssc registered [ 1806.051231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.932154] Key type lgssc unregistered [ 1835.255890] LNet: 99204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.283273] LNetError: 99204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.305129] LNet: Removed LNI 192.168.206.55@tcp [ 1835.902636] Key type .llcrypt unregistered [ 1835.904104] Key type ._llcrypt unregistered [ 1853.922448] Key type ._llcrypt registered [ 1853.928639] Key type .llcrypt registered [ 1855.352253] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:45:52 (1787550352) [ 1855.653744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.669501] alg: No test for adler32 (adler32-zlib) [ 1856.591524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.706295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1856.813130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1856.875842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1856.939855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1864.229537] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:46:01 (1787550361) [ 1864.421910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.481331] LNet: 100620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.514417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.590540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1864.668232] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1865.684344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.719868] LNet: 100764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.725492] LNetError: 100764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.754325] LNet: Removed LNI 192.168.206.55@tcp [ 1865.784571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.851421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1865.890464] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1866.618790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.651213] LNetError: 100792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.664407] LNet: Removed LNI 192.168.206.55@tcp [ 1866.703666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.764780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1866.807439] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1876.097442] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:46:13 (1787550373) [ 1876.244275] LNet: 101375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.254051] LNet: 101375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1876.281107] LNetError: 101375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.316796] LNet: Removed LNI 192.168.206.55@tcp [ 1877.182850] Key type .llcrypt unregistered [ 1877.188810] Key type ._llcrypt unregistered [ 1883.351974] Key type ._llcrypt registered [ 1883.355720] Key type .llcrypt registered [ 1883.508841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1883.524352] alg: No test for adler32 (adler32-zlib) [ 1884.441272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.561731] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1884.564727] LNet: *** cfs_fail_loc=e003, val=100*** [ 1885.226471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1885.253467] LNet: *** cfs_fail_loc=e003, val=100*** [ 1885.256949] LNet: Skipped 2 previous similar messages [ 1885.262530] Ping reply received with mapping: [ 1885.264703] found metadata reply NI[192.168.206.55@tcp] [ 1885.270569] 37 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.276322] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.280557] 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.465767] LNet: 101784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.486722] LNetError: 101784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.516468] LNet: Removed LNI 192.168.206.55@tcp [ 1886.326762] Key type .llcrypt unregistered [ 1886.334113] Key type ._llcrypt unregistered [ 1898.389703] Key type ._llcrypt registered [ 1898.391580] Key type .llcrypt registered [ 1899.861777] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:46:37 (1787550397) [ 1900.416243] Key type .llcrypt unregistered [ 1900.420462] Key type ._llcrypt unregistered [ 1906.781754] Key type ._llcrypt registered [ 1906.783449] Key type .llcrypt registered [ 1907.076819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.112315] alg: No test for adler32 (adler32-zlib) [ 1908.312606] LNetError: 102841:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1908.838353] Key type .llcrypt unregistered [ 1908.841142] Key type ._llcrypt unregistered [ 1914.881725] Key type ._llcrypt registered [ 1914.885911] Key type .llcrypt registered [ 1915.904647] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:46:53 (1787550413) [ 1916.584294] Key type .llcrypt unregistered [ 1916.588112] Key type ._llcrypt unregistered [ 1923.037090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1923.039610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1925.550797] Key type ._llcrypt registered [ 1925.555137] Key type .llcrypt registered [ 1925.740570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1925.757452] alg: No test for adler32 (adler32-zlib) [ 1926.667599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.764941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1936.241588] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:47:13 (1787550433) [ 1936.368756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.395127] LNet: 104629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.411860] LNetError: 104629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1936.430565] LNet: Removed LNI 10.1.2.3@tcp [ 1936.458711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.338599] Key type .llcrypt unregistered [ 1937.344134] Key type ._llcrypt unregistered [ 1944.052995] Key type ._llcrypt registered [ 1944.057496] Key type .llcrypt registered [ 1944.328551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1944.349899] alg: No test for adler32 (adler32-zlib) [ 1945.335211] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1945.435996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1945.675654] LNet: 104923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.682302] LNetError: 104923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.708772] LNet: Removed LNI 192.168.206.55@tcp [ 1946.513615] Key type .llcrypt unregistered [ 1946.515330] Key type ._llcrypt unregistered [ 1960.589338] Key type ._llcrypt registered [ 1960.592914] Key type .llcrypt registered [ 1962.365293] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:47:39 (1787550459) [ 1963.247194] Key type .llcrypt unregistered [ 1963.251246] Key type ._llcrypt unregistered [ 1969.954288] Key type ._llcrypt registered [ 1969.958375] Key type .llcrypt registered [ 1970.127631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1970.141964] alg: No test for adler32 (adler32-zlib) [ 1979.868503] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:47:56 (1787550476) [ 1980.033783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.117455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.248548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.287951] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1981.203846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.259727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1981.322176] LNet: Removed LNI 192.168.206.55@tcp [ 1981.713941] LNet: 106679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.214449] Key type .llcrypt unregistered [ 1982.221432] Key type ._llcrypt unregistered [ 1993.901897] Key type ._llcrypt registered [ 1993.904625] Key type .llcrypt registered [ 1995.599705] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:48:12 (1787550492) [ 1996.131317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.144727] alg: No test for adler32 (adler32-zlib) [ 1997.162973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.266273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.303038] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 1998.375101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 1998.448220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.55@tcp [ 1998.483803] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1998.491821] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.506756] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 0 [ 1998.513450] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1998.517483] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1998.522549] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.525450] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 0 [ 1998.529500] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1998.532856] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1998.536479] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.539719] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 2048 [ 1998.544411] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1998.548468] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1998.553457] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1998.562585] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1998, 0, 2048 [ 1998.572607] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2005.260885] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:48:22 (1787550502) [ 2005.775427] LNet: 108219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.786762] LNetError: 108219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.808386] LNet: Removed LNI 192.168.206.55@tcp [ 2006.503878] Key type .llcrypt unregistered [ 2006.505872] Key type ._llcrypt unregistered [ 2012.234922] Key type ._llcrypt registered [ 2012.238872] Key type .llcrypt registered [ 2012.384491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2012.402668] alg: No test for adler32 (adler32-zlib) [ 2013.332683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2013.362607] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2022.224466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 2030.593650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.626960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2030.655940] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 2031.501488] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2031.623404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2031.636885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2031.646599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2033.979512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2034.014282] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2043.787726] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:49:01 (1787550541) [ 2043.946406] LNet: 109727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.955826] LNetError: 109727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.965560] LNet: Removed LNI 192.168.206.55@tcp [ 2043.967796] LNet: Removed LNI 192.168.206.55@tcp1 [ 2043.990957] LNet: Removed LNI 10.1.2.3@tcp99 [ 2044.647909] Key type .llcrypt unregistered [ 2044.653375] Key type ._llcrypt unregistered [ 2050.992817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2050.995577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2053.553353] Key type ._llcrypt registered [ 2053.556575] Key type .llcrypt registered [ 2053.775159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2053.795629] alg: No test for adler32 (adler32-zlib) [ 2054.702711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.785481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.839887] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2055.496772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2055.525950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2057.532485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2057.573981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2057.620734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2057.640945] LNetError: 110482:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2057.670314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2057.702726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2057.987132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2058.031538] LNet: 110502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.038463] LNetError: 110502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2058.043748] LNet: Removed LNI 10.1.2.3@tcp [ 2059.110466] LNet: Removed LNI 192.168.206.55@tcp [ 2059.674601] Key type .llcrypt unregistered [ 2059.676358] Key type ._llcrypt unregistered [ 2072.388140] Key type ._llcrypt registered [ 2072.395320] Key type .llcrypt registered [ 2074.367683] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:49:31 (1787550571) [ 2074.951343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.961077] alg: No test for adler32 (adler32-zlib) [ 2076.050614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.141990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.194627] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2077.399515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2077.503489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg655-client@tcp [ 2086.369916] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:49:43 (1787550583) [ 2086.715221] LNet: 112042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.729423] LNetError: 112042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.746056] LNet: Removed LNI 192.168.206.55@tcp [ 2087.933389] Key type .llcrypt unregistered [ 2087.936581] Key type ._llcrypt unregistered [ 2097.024478] Key type ._llcrypt registered [ 2097.025847] Key type .llcrypt registered [ 2097.172766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2097.192713] alg: No test for adler32 (adler32-zlib) [ 2098.138970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2098.157050] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2098.175291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2105.891389] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:50:03 (1787550603) [ 2107.397028] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2109.074241] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:50:06 (1787550606) [ 2110.543177] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2112.036233] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:50:09 (1787550609) [ 2112.340422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.370217] LNet: 113468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.378429] LNetError: 113468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.398824] LNet: Removed LNI 192.168.206.55@tcp [ 2112.449891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.536837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.571308] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2113.339532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2113.402890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2119.470719] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:50:16 (1787550616) [ 2119.814930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.842824] LNet: 114178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.866102] LNetError: 114178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.894046] LNet: Removed LNI 192.168.206.55@tcp [ 2119.915716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.971808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2120.004639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2120.546572] Key type .llcrypt unregistered [ 2120.549695] Key type ._llcrypt unregistered [ 2132.213516] Key type ._llcrypt registered [ 2132.216149] Key type .llcrypt registered [ 2133.624251] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:50:30 (1787550630) [ 2134.318923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.339652] alg: No test for adler32 (adler32-zlib) [ 2135.234799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.292478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.336321] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2136.240868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2136.324657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2144.663677] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:50:41 (1787550641) [ 2144.792684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.840349] LNet: 115729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.854992] LNetError: 115729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.883706] LNet: Removed LNI 192.168.206.55@tcp [ 2144.950829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.047742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2145.108324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2153.778719] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:50:50 (1787550650) [ 2153.837869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.855468] LNet: 116314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.877945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.925663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2154.350027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2154.394472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2154.481821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2155.092916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2155.169258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2155.272393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.321213] LNet: 116352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.387091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.461757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2156.122756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2156.195862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2156.300894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2156.920883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2156.971115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2165.034800] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:51:02 (1787550662) [ 2165.289198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.320615] LNet: 116953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.411831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.499637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.520811] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2166.372181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2166.396225] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2167.409626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2167.480541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2167.679431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.733824] LNetError: 117232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.744592] LNet: Removed LNI 192.168.206.55@tcp [ 2167.763409] LNet: Removed LNI 192.168.206.55@tcp2 [ 2167.816116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.910109] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2167.918102] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2167.932055] LNet: 117238:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2168.590195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2168.797988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.818142] LNetError: 117275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.824080] LNetError: 117275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2168.830719] LNet: Removed LNI 192.168.206.55@tcp [ 2168.843268] LNet: Removed LNI 192.168.206.55@tcp2 [ 2168.868954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.910552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.928309] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2169.643099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2169.666703] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2170.345516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2170.383763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2170.435318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2170.630744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.647383] LNet: 117560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.652918] LNet: 117560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2171.680464] LNet: Removed LNI 192.168.206.55@tcp [ 2171.688204] LNetError: 117560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.706858] LNetError: 117560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2171.740861] LNet: Removed LNI 192.168.206.55@tcp2 [ 2171.791747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.843895] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2171.852854] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2171.862895] LNet: 117566:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2172.526407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2172.834323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.877745] LNet: Removed LNI 192.168.206.55@tcp [ 2173.933263] LNet: Removed LNI 192.168.206.55@tcp2 [ 2173.979810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.055201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2174.091562] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2174.896452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2174.929505] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2175.667655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2175.723210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2175.951326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.989882] LNetError: 117882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.995584] LNetError: 117882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2176.013840] LNet: Removed LNI 192.168.206.55@tcp [ 2176.028959] LNet: Removed LNI 192.168.206.55@tcp2 [ 2176.097184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.191404] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2176.199485] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2176.207236] LNet: 117888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2176.549034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2176.771100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2177.823348] LNet: Removed LNI 192.168.206.55@tcp [ 2177.835146] LNet: Removed LNI 192.168.206.55@tcp2 [ 2177.886591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.971614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.026734] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2178.884969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2178.921510] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2179.809358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2179.853792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2179.909442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2180.184734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.220474] LNet: 118210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.227824] LNet: 118210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2181.280025] LNet: Removed LNI 192.168.206.55@tcp [ 2182.312277] LNet: Removed LNI 192.168.206.55@tcp2 [ 2182.406291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.473167] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2182.485043] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2182.492804] LNet: 118217:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2183.241793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2183.681418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.716405] LNet: Removed LNI 192.168.206.55@tcp [ 2184.744518] LNet: Removed LNI 192.168.206.55@tcp2 [ 2184.786287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.855421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.907427] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2186.014793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2186.049956] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2187.714730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2187.744968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2187.808554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2188.468571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.531919] LNetError: 118551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.550163] LNetError: 118551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2188.564895] LNet: Removed LNI 192.168.206.55@tcp [ 2188.605473] LNet: Removed LNI 192.168.206.55@tcp2 [ 2188.734506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.874491] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2188.888388] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2188.897515] LNet: 118557:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2189.994550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2190.344929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.391145] LNet: Removed LNI 192.168.206.55@tcp [ 2191.410562] LNet: Removed LNI 192.168.206.55@tcp2 [ 2191.507730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.631575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.672985] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2192.844575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2192.888753] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2194.823258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2194.910440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2194.977130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2195.112837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2195.700786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.769045] LNet: Removed LNI 192.168.206.55@tcp [ 2197.868220] LNet: Removed LNI 192.168.206.55@tcp2 [ 2197.972129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.120961] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2198.129856] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2198.139142] LNet: 118918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.896415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2210.516103] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2212.868124] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:51:49 (1787550709) [ 2213.031904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.051695] LNet: 119677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.059585] LNet: 119677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2213.072354] LNetError: 119677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.081714] LNetError: 119677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2213.100550] LNet: Removed LNI 192.168.206.55@tcp [ 2213.126899] LNet: Removed LNI 192.168.206.55@tcp2 [ 2213.152184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.330925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2213.346941] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2214.362762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2214.404825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2215.745293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.761639] LNet: Removed LNI 10.1.2.3@tcp [ 2215.778819] LNet: Removed LNI 192.168.206.55@tcp [ 2215.800644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.871580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2215.943684] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2215.955526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2216.996638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2217.013695] LNet: Removed LNI 192.168.206.55@tcp [ 2217.032790] LNet: Removed LNI 10.1.2.3@tcp [ 2223.996732] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:52:01 (1787550721) [ 2224.216029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.311414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.384258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.402538] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2225.290206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2225.318512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.348484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.396694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.428255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.456680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.512796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.558899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.601660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.639274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.665656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.55@tcp [ 2225.833706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2233.236882] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:52:10 (1787550730) [ 2233.381687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.471046] LNet: Removed LNI 192.168.206.55@tcp [ 2234.515778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.646845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2234.790670] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2234.805812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2234.821025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2237.531784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.571263] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2238.525201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2238.549635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2241.758988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2241.786505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2241.846760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.55@tcp -m PUT -e local_timeout -r 1 [ 2241.916407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2241.965773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2241.971796] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2242.444746] LNetError: 121358:0:(lib-move.c:4801:lnet_parse()) 192.168.206.55@tcp, src 192.168.206.55@tcp: Dropping PUT (error -108 looking up sender) [ 2243.491752] LNet: Removed LNI 10.1.2.3@tcp [ 2244.522091] LNet: Removed LNI 192.168.206.55@tcp [ 2245.344343] Key type .llcrypt unregistered [ 2245.350686] Key type ._llcrypt unregistered [ 2257.945938] Key type ._llcrypt registered [ 2257.948505] Key type .llcrypt registered [ 2259.328973] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:52:36 (1787550756) [ 2259.853607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.879481] alg: No test for adler32 (adler32-zlib) [ 2260.824857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.240565] LNet: 122674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.965435] Key type .llcrypt unregistered [ 2261.971419] Key type ._llcrypt unregistered [ 2269.048904] Key type ._llcrypt registered [ 2269.051269] Key type .llcrypt registered [ 2269.285641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2269.306252] alg: No test for adler32 (adler32-zlib) [ 2270.288982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2270.319119] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2278.727118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 2278.993597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2279.725069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.155@tcp [ 2279.952516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2279.979045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2280.003595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2280.033369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2280.076916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2280.124584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2280.186431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2280.245488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2280.274442] LNet: 123128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2280.305167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2280.331986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2280.369954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2280.404435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2280.449397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2280.493599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2281.353234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2281.567167] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.155@tcp [ 2281.740873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2281.800457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2281.834712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2281.861636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2281.891770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2281.935693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2281.961361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2282.011631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2282.038802] LNet: 123245:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2282.075543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2282.136567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2282.165274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2282.206675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2282.238212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2282.266812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2282.979482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.155@tcp [ 2283.196478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2283.237025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2283.279807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2283.316736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2283.354469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2283.395680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2283.436420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2283.490338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2283.516817] LNet: 123362:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.533097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2283.592310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2283.628552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2283.673568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2283.712883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2283.768857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2284.389625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2284.602233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.632451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2284.655291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2284.693474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2284.724211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2284.746472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2284.774636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2284.803133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2284.829812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2284.867837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2284.896088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2284.926631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2284.968858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2284.994679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2285.285701] LNet: 123530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.306574] LNetError: 123530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.325801] LNet: Removed LNI 192.168.206.55@tcp [ 2285.829420] Key type .llcrypt unregistered [ 2285.831826] Key type ._llcrypt unregistered [ 2303.952509] Key type ._llcrypt registered [ 2303.958965] Key type .llcrypt registered [ 2306.424505] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:53:22 (1787550802) [ 2307.360380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2307.413871] alg: No test for adler32 (adler32-zlib) [ 2308.635472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.311785] LNet: 124420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.321424] Key type .llcrypt unregistered [ 2310.333120] Key type ._llcrypt unregistered [ 2319.837618] Key type ._llcrypt registered [ 2319.839715] Key type .llcrypt registered [ 2320.350584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2320.381207] alg: No test for adler32 (adler32-zlib) [ 2321.496579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2321.565604] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2331.875140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.155@tcp [ 2342.885615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.980044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2343.035089] LNet: Added LNI 192.168.206.55@tcp1 [8/256/0/180] [ 2344.646724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2345.597489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.155@tcp [ 2345.870451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2345.900255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2345.931685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2345.969690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2346.011607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2346.047596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2346.078545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2346.108512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2346.125888] LNet: 125080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2346.145656] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.155@tcp [ 2346.148299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2346.188171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2346.240425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2346.279176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2346.326204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2346.354249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2347.092674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2347.338651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2347.378360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2347.433494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2347.474086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.527665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.579615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.651053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.709821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.737600] LNet: 125197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.774732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.834126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.872761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.920851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.971834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2348.004771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2348.702655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.155@tcp [ 2348.978787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2349.009917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2349.043387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2349.083662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2349.160629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2349.206137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2349.265773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2349.337850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2349.372654] LNet: 125314:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2349.414646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2349.477583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2349.514196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2349.572555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2349.619279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2349.656533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2350.304289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2350.549765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2350.573617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2350.606842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2350.634252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2350.669646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2350.702695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2350.743570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2350.770859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2350.813688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2350.839835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2350.863267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2350.893246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2350.922172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2350.963450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.870548] LNet: 125516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.883839] LNetError: 125516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.895627] LNet: Removed LNI 192.168.206.55@tcp [ 2356.923962] LNet: Removed LNI 192.168.206.55@tcp1 [ 2357.709504] Key type .llcrypt unregistered [ 2357.714092] Key type ._llcrypt unregistered [ 2379.444323] Key type ._llcrypt registered [ 2379.450733] Key type .llcrypt registered [ 2381.039900] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:54:38 (1787550878) [ 2381.435987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.457906] alg: No test for adler32 (adler32-zlib) [ 2382.467754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.534422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.567335] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2383.400655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2383.444477] LNet: Added LNI 192.168.206.55@tcp2 [8/256/0/180] [ 2384.508933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.55@tcp [ 2384.623804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2384.660870] LNet: Removed LNI 192.168.206.55@tcp [ 2392.464742] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:54:49 (1787550889) [ 2394.381934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2396.085344] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:54:53 (1787550893) [ 2403.377625] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:55:00 (1787550900) [ 2411.065068] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:55:08 (1787550908) [ 2417.777957] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:55:15 (1787550915) [ 2424.518893] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:55:21 (1787550921) [ 2430.945674] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:55:28 (1787550928) [ 2437.944291] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:55:35 (1787550935) [ 2446.174511] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:55:43 (1787550943) [ 2455.885367] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:55:52 (1787550952) [ 2466.004374] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:56:03 (1787550963) [ 2474.659695] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:56:11 (1787550971) [ 2483.611625] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:56:20 (1787550980) [ 2493.441752] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:56:30 (1787550990) [ 2500.326275] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:56:37 (1787550997) [ 2509.434534] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:56:46 (1787551006) [ 2516.465203] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:56:54 (1787551014) [ 2523.506540] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:57:01 (1787551021) [ 2530.294188] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:57:07 (1787551027) [ 2530.389168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2530.421637] LNet: 137204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.432421] LNetError: 137204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.459588] LNet: Removed LNI 192.168.206.55@tcp2 [ 2530.507091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.640156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2530.662047] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2531.888864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2531.911393] LNet: 137354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.916231] LNetError: 137354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.934015] LNet: Removed LNI 192.168.206.55@tcp [ 2531.954777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.016134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2532.040800] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2537.802635] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:57:15 (1787551035) [ 2543.993319] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:57:21 (1787551041) [ 2550.517071] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:57:28 (1787551048) [ 2557.545412] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:57:35 (1787551055) [ 2564.049991] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:57:41 (1787551061) [ 2570.895547] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:57:48 (1787551068) [ 2579.213845] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:57:55 (1787551075) [ 2589.896719] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:58:06 (1787551086) [ 2601.582839] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:58:18 (1787551098) [ 2611.018548] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:58:28 (1787551108) [ 2619.846833] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:58:37 (1787551117) [ 2628.646516] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:58:45 (1787551125) [ 2638.171666] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2404 sec ========== 01:58:55 (1787551135) [ 2638.607547] LNet: 144820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.618513] LNetError: 144820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.629307] LNet: Removed LNI 192.168.206.55@tcp [ 2639.391675] Key type .llcrypt unregistered [ 2639.397697] Key type ._llcrypt unregistered [ 2654.408721] Key type ._llcrypt registered [ 2654.426397] Key type .llcrypt registered [ 2654.775233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2654.792734] alg: No test for adler32 (adler32-zlib) [ 2655.856944] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 2656.065161] LNet: Added LNI 192.168.206.55@tcp [8/256/0/180] [ 2657.714472] Key type lgssc registered [ 2658.762619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2724.396414] Lustre: Mounted lustre-client - version 2.17.57_82_g53f9976 [ 2731.738581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2744.242355] Lustre: Unmounted lustre-client [ 2771.633686] Key type lgssc unregistered [ 2772.005671] LNet: 147793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.021179] LNetError: 147793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.051813] LNet: Removed LNI 192.168.206.55@tcp [ 2772.912151] Key type .llcrypt unregistered [ 2772.914132] Key type ._llcrypt unregistered