[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 429274486 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003317] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007985] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010120] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.020708] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025092] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.030435] *** VALIDATE proc *** [ 0.032098] *** VALIDATE cgroup *** [ 0.032922] *** VALIDATE cgroup2 *** [ 0.034185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041435] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052276] rcu: Hierarchical SRCU implementation. [ 0.054547] smp: Bringing up secondary CPUs ... [ 0.055547] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059404] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132522] node 0 deferred pages initialised in 69ms [ 0.136274] devtmpfs: initialized [ 0.137297] x86/mm: Memory block size: 128MB [ 0.139875] gcov: version magic: 0x41383552 [ 0.141364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147318] pinctrl core: initialized pinctrl subsystem [ 0.149138] [ 0.149647] ************************************************************* [ 0.151016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ** ** [ 0.155013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157011] ** ** [ 0.159010] ** This means that this kernel is built to expose internal ** [ 0.161011] ** IOMMU data structures, which may compromise security on ** [ 0.163011] ** your system. ** [ 0.165012] ** ** [ 0.168011] ** If you see this message and you are not debugging the ** [ 0.169009] ** kernel, report this immediately to your vendor! ** [ 0.171010] ** ** [ 0.173012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175010] ************************************************************* [ 0.177655] NET: Registered protocol family 16 [ 0.179436] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188481] cpuidle: using governor menu [ 0.189565] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193077] PCI: Using configuration type 1 for base access [ 0.196126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211049] cryptd: max_cpu_qlen set to 1000 [ 0.213143] ACPI: Added _OSI(Module Device) [ 0.214014] ACPI: Added _OSI(Processor Device) [ 0.216011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222266] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228669] ACPI: Interpreter enabled [ 0.230060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232013] ACPI: Using IOAPIC for interrupt routing [ 0.234090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237233] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.247689] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260257] acpiphp: Slot [2] registered [ 0.261105] acpiphp: Slot [5] registered [ 0.263114] acpiphp: Slot [6] registered [ 0.264222] acpiphp: Slot [3] registered [ 0.266135] acpiphp: Slot [4] registered [ 0.267089] acpiphp: Slot [7] registered [ 0.268112] acpiphp: Slot [8] registered [ 0.270122] acpiphp: Slot [9] registered [ 0.271121] acpiphp: Slot [10] registered [ 0.273190] acpiphp: Slot [11] registered [ 0.274127] acpiphp: Slot [12] registered [ 0.276130] acpiphp: Slot [13] registered [ 0.277075] acpiphp: Slot [14] registered [ 0.278118] acpiphp: Slot [15] registered [ 0.280091] acpiphp: Slot [16] registered [ 0.281096] acpiphp: Slot [17] registered [ 0.282100] acpiphp: Slot [18] registered [ 0.284151] acpiphp: Slot [19] registered [ 0.286142] acpiphp: Slot [20] registered [ 0.288107] acpiphp: Slot [21] registered [ 0.289089] acpiphp: Slot [22] registered [ 0.291079] acpiphp: Slot [23] registered [ 0.292104] acpiphp: Slot [24] registered [ 0.294096] acpiphp: Slot [25] registered [ 0.295117] acpiphp: Slot [26] registered [ 0.297095] acpiphp: Slot [27] registered [ 0.298095] acpiphp: Slot [28] registered [ 0.299103] acpiphp: Slot [29] registered [ 0.301083] acpiphp: Slot [30] registered [ 0.303109] acpiphp: Slot [31] registered [ 0.304069] PCI host bridge to bus 0000:00 [ 0.305019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.318023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326313] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.338053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343047] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348381] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351970] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354109] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356745] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381410] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444686] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.492351] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.494361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.499217] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.502024] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.504257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.509037] iommu: Default domain type: Passthrough [ 0.510271] SCSI subsystem initialized [ 0.511169] ACPI: bus type USB registered [ 0.513111] usbcore: registered new interface driver usbfs [ 0.515077] usbcore: registered new interface driver hub [ 0.516076] usbcore: registered new device driver usb [ 0.518165] pps_core: LinuxPPS API ver. 1 registered [ 0.520010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.523053] PTP clock support registered [ 0.526164] EDAC MC: Ver: 3.0.0 [ 0.527361] PCI: Using ACPI for IRQ routing [ 0.528779] NetLabel: Initializing [ 0.529008] NetLabel: domain hash size = 128 [ 0.530008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.531157] NetLabel: unlabeled traffic allowed by default [ 0.533159] vgaarb: loaded [ 0.535250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.537011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.543288] clocksource: Switched to clocksource kvm-clock [ 0.649142] VFS: Disk quotas dquot_6.6.0 [ 0.650572] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.653088] *** VALIDATE ramfs *** [ 0.654422] *** VALIDATE hugetlbfs *** [ 0.656423] pnp: PnP ACPI init [ 0.658813] pnp: PnP ACPI: found 6 devices [ 0.687385] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.690640] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.692414] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.695022] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.697483] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.699957] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.702786] NET: Registered protocol family 2 [ 0.705307] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.709543] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.712875] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.718460] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.721889] TCP: Hash tables configured (established 65536 bind 65536) [ 0.725000] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.727266] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.730247] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.732650] NET: Registered protocol family 1 [ 0.735696] RPC: Registered named UNIX socket transport module. [ 0.737517] RPC: Registered udp transport module. [ 0.739128] RPC: Registered tcp transport module. [ 0.740657] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.742705] NET: Registered protocol family 44 [ 0.744197] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.746573] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.748582] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.750628] PCI: CLS 0 bytes, default 64 [ 0.751782] Unpacking initramfs... [ 2.169702] debug: unmapping init [mem 0xffff99dcfcc64000-0xffff99dcfffcffff] [ 2.174227] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.176895] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.180079] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.674775] Initialise system trusted keyrings [ 2.677956] Key type blacklist registered [ 2.680138] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.689272] zbud: loaded [ 2.692347] *** VALIDATE nfs *** [ 2.693691] *** VALIDATE nfs4 *** [ 2.696037] pstore: using deflate compression [ 2.700765] Platform Keyring initialized [ 2.823524] NET: Registered protocol family 38 [ 2.825549] Key type asymmetric registered [ 2.827495] Asymmetric key parser 'x509' registered [ 2.829709] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.833245] io scheduler mq-deadline registered [ 2.835170] io scheduler kyber registered [ 2.837140] io scheduler bfq registered [ 2.839257] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.842574] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.847716] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.850692] ACPI: Power Button [PWRF] [ 2.855856] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.862644] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.878816] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.906858] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.935030] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.942978] Non-volatile memory driver v1.3 [ 2.944848] Linux agpgart interface v0.103 [ 2.978400] virtio_blk virtio1: [vda] 149392 512-byte logical blocks (76.5 MB/72.9 MiB) [ 2.981502] vda: detected capacity change from 0 to 76488704 [ 2.998260] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.002191] vdb: detected capacity change from 0 to 1073741824 [ 3.011252] libphy: Fixed MDIO Bus: probed [ 3.018286] usbcore: registered new interface driver usbserial_generic [ 3.020749] usbserial: USB Serial support registered for generic [ 3.023339] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.027597] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.029784] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.035074] mousedev: PS/2 mouse device common for all mice [ 3.038207] rtc_cmos 00:05: RTC can wake from S4 [ 3.041111] rtc_cmos 00:05: registered as rtc0 [ 3.042113] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.043147] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.049650] intel_pstate: CPU model not supported [ 3.052950] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.057166] hid: raw HID events driver (C) Jiri Kosina [ 3.059531] usbcore: registered new interface driver usbhid [ 3.061972] usbhid: USB HID core driver [ 3.064196] drop_monitor: Initializing network drop monitor service [ 3.064225] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.066697] Initializing XFRM netlink socket [ 3.071099] NET: Registered protocol family 10 [ 3.074157] Segment Routing with IPv6 [ 3.075683] NET: Registered protocol family 17 [ 3.077884] mpls_gso: MPLS GSO support [ 3.084701] RAS: Correctable Errors collector initialized. [ 3.087102] AVX version of gcm_enc/dec engaged. [ 3.089178] AES CTR mode by8 optimization enabled [ 3.164255] sched_clock: Marking stable (3164234638, 0)->(4041742540, -877507902) [ 3.168225] registered taskstats version 1 [ 3.171100] Loading compiled-in X.509 certificates [ 3.173101] zswap: loaded using pool lzo/zbud [ 3.196303] Key type big_key registered [ 3.215791] Key type encrypted registered [ 3.217423] ima: No TPM chip found, activating TPM-bypass! [ 3.219678] ima: Allocated hash algorithm: sha1 [ 3.220867] ima: No architecture policies found [ 3.221887] evm: Initialising EVM extended attributes: [ 3.223396] evm: security.selinux [ 3.224273] evm: security.ima [ 3.224954] evm: security.capability [ 3.225871] evm: HMAC attrs: 0x1 [ 3.228365] rtc_cmos 00:05: setting system clock to 2026-08-22 10:47:08 UTC (1787395628) [ 3.234203] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.237833] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 3.247224] Write protecting the kernel read-only data: 28672k [ 3.250794] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 3.253618] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 3.288769] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.298578] systemd[1]: Detected virtualization kvm. [ 3.300693] systemd[1]: Detected architecture x86-64. [ 3.302899] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.328203] systemd[1]: No hostname configured. [ 3.330174] systemd[1]: Set hostname to . [ 3.332596] random: systemd: uninitialized urandom read (16 bytes read) [ 3.335057] systemd[1]: Initializing machine ID from random generator. [ 3.478699] random: systemd: uninitialized urandom read (16 bytes read) [ 3.480956] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.486291] random: systemd: uninitialized urandom read (16 bytes read) [ 3.488400] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.493138] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.177313] device-mapper: uevent: version 1.0.3 [ 4.179221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.858753] virtio_net virtio0 ens2: renamed from eth0 [ 4.894831] scsi host0: ata_piix [ 4.896507] scsi host1: ata_piix [ 4.897816] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.899540] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.088474] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.279401] random: fast init done [ 9.733041] random: crng init done [ 9.734481] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.016229] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.096954] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.363203] SELinux: Disabled at runtime. [ 11.416816] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.426304] systemd[1]: Detected virtualization kvm. [ 11.428262] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.954852] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.959373] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.964824] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.969682] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.972411] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.978544] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.982311] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ 12.049945] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.390740] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.726536] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.774246] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.839852] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.863414] EDAC sbridge: Ver: 1.1.2 [ 14.025042] Key type dns_resolver registered [ 14.344913] NFS: Registering the id_resolver key type [ 14.346440] Key type id_resolver registered [ 14.347547] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg610-client login: [ 59.330575] libcfs: loading out-of-tree module taints kernel. [ 59.470600] Key type ._llcrypt registered [ 59.474196] Key type .llcrypt registered [ 60.109711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 60.127321] alg: No test for adler32 (adler32-zlib) [ 61.744532] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 63.015466] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 64.855148] Key type lgssc registered [ 66.831641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.250588] hrtimer: interrupt took 4363797 ns [ 197.899908] Lustre: Mounted lustre-client [ 202.968197] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.679986] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing check_logdir /tmp/testlogs/ [ 223.711970] Lustre: lustre-OST0000-osc-ffff99dd44c8c800: disconnect after 23s idle [ 223.812230] Lustre: DEBUG MARKER: oleg610-client.virtnet: executing yml_node [ 227.956579] Lustre: DEBUG MARKER: Client: 2.17.54.108 [ 230.650615] Lustre: DEBUG MARKER: MDS: 2.17.54.108 [ 233.132928] Lustre: DEBUG MARKER: OSS: 2.17.54.108 [ 234.821912] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 06:50:58 EDT 2026 [ 250.456313] Lustre: Unmounted lustre-client [ 278.364160] Key type lgssc unregistered [ 278.594926] LNet: 8389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.611738] LNetError: 8389:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.637645] LNet: Removed LNI 192.168.206.10@tcp [ 279.262175] Key type .llcrypt unregistered [ 279.265723] Key type ._llcrypt unregistered [ 306.648477] Key type ._llcrypt registered [ 306.651051] Key type .llcrypt registered [ 306.679676] LNet: 8968:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 306.679676] mask usage: [+|-] ... [ 306.898597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.909505] alg: No test for adler32 (adler32-zlib) [ 307.840834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 307.944929] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 308.010828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.030068] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.418752] LNet: 9273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.425411] LNetError: 9273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.451125] LNet: Removed LNI 192.168.206.10@tcp [ 313.107386] Key type .llcrypt unregistered [ 313.111520] Key type ._llcrypt unregistered [ 319.426502] Key type ._llcrypt registered [ 319.429314] Key type .llcrypt registered [ 321.585585] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:52:24 (1787395944) [ 321.777890] LNet: 9643:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 321.777890] mask usage: [+|-] ... [ 321.995041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 322.022884] alg: No test for adler32 (adler32-zlib) [ 323.004497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.099128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.112594] LNet: 9684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.126510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.173633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 323.189493] LNet: 9691:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 330.484430] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:52:34 (1787395954) [ 330.561963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.582217] LNet: 10261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.606670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.686547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 338.384131] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:52:42 (1787395962) [ 338.477396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.523764] LNet: 10848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.555880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.631539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.241147] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:52:49 (1787395969) [ 345.412599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.431356] LNet: 11438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.451509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.502347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 351.374695] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:52:55 (1787395975) [ 351.707063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.736652] LNet: 12024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.765512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.875282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 352.400576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 359.794387] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:53:03 (1787395983) [ 360.134865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.175466] LNet: 12616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.200027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.273844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.851680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 361.201909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 368.924463] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:53:12 (1787395992) [ 369.048781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.148771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.293314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 377.656743] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:53:21 (1787396001) [ 377.800627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.831807] LNet: 13791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.839498] LNet: 13791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 377.865684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.980956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 378.016694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.556754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 378.595341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 378.931994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 378.966656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 379.355212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 379.393205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 379.703957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 379.725881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 380.026226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 380.051125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.341674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 380.362825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.614354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 380.641283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 380.899772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 380.925346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 381.193266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 381.239852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 381.631197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 381.671305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 382.174493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 382.229529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 382.548395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 382.596608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.767749] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:53:33 (1787396013) [ 389.930369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.034619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.180112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 390.237411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 396.342464] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:53:40 (1787396020) [ 396.505419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.573516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.694665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 396.762461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 403.741961] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:53:47 (1787396027) [ 403.928533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.970901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.093793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 404.135710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 410.441843] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:53:54 (1787396034) [ 410.668300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.686312] LNet: 16237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.690910] LNet: 16237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 410.721843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.815443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 410.852658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 417.809418] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:54:01 (1787396041) [ 418.103557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.152478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.215249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 418.237628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 424.779775] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:54:08 (1787396048) [ 425.046665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.085602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.149665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 425.190444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 431.811765] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:54:15 (1787396055) [ 431.913262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.981651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.104743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.609400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.050470] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:54:22 (1787396062) [ 439.186068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.253745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.469222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.681454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.754641] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:54:30 (1787396070) [ 446.853304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.913536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.107623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.376412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.095303] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:54:38 (1787396078) [ 454.345914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.405969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.828482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.146157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.089786] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:54:45 (1787396085) [ 462.302260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.339953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.403415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.490342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.652412] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:54:53 (1787396093) [ 470.040502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.095468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.155655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.238936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.559785] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:55:01 (1787396101) [ 477.757186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.787749] LNet: 21677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.794725] LNet: 21677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 477.820297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.916964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 484.879352] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:55:08 (1787396108) [ 485.147737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.206670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.290810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 492.710389] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:55:16 (1787396116) [ 492.948731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.985486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.054890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 493.402866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 499.811529] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:55:23 (1787396123) [ 499.983774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.032864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.103531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 500.137948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 507.072448] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:55:31 (1787396131) [ 507.147792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.189880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.292815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 507.321874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 514.099923] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:55:37 (1787396137) [ 514.240052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.319463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.399789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 514.442576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 521.604358] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:55:45 (1787396145) [ 521.812891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.834331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.884786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.943079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.803026] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:55:52 (1787396152) [ 529.103764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.160390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.238078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 529.314253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 529.340793] LNetError: 25792:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 529.380786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 529.414637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 535.816419] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:55:59 (1787396159) [ 535.946948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.989863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.062607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gIWT3lB0mX [ 536.131966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gIWT3lB0mX [ 536.195381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gIWT3lB0mX [ 543.192989] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:56:06 (1787396166) [ 543.296877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.376968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.464187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.508784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.568546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.463268] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:56:13 (1787396173) [ 549.820752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.851950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.931436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 552.472128] LNet: Message forwarding will be enabled in 2s [ 555.551071] LNet: Message forwarding enabled [ 555.960190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.009404] LNet: Message forwarding will stop in 2s to 6s [ 560.528383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.825350] LNet: Message forwarding disabled [ 562.044333] LNet: Message forwarding will be enabled in 2s [ 565.984633] LNet: Message forwarding enabled [ 568.610638] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:56:32 (1787396192) [ 568.734330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.038929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.163380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 576.591762] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:56:40 (1787396200) [ 576.835490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.884510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.928725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 576.956578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 577.001982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 577.032238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 577.086474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 577.137455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 577.167423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.192982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 577.243202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 577.281278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 577.310615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 577.362952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 577.403503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 577.435884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 577.469640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 577.542995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 577.597445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 577.649730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.695267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 577.719897] LNetError: 28861:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 584.139990] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:56:48 (1787396208) [ 584.359292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.447231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.543169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.456206] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:56:55 (1787396215) [ 593.121527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg610-server [ 594.749593] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:56:58 (1787396218) [ 595.059590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.086220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.131426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.163814] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 596.572600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[11-15]@tcp [ 596.600837] LNet: 30577:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 596.650991] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.12@tcp is being used as a gateway but routing feature is not turned on [ 596.677505] LNetError: 30367:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.12@tcp has gone from up to down [ 597.673164] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.12@tcp is being used as a gateway but routing feature is not turned on [ 597.695210] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 598.690315] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 598.704597] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 600.739947] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.14@tcp is being used as a gateway but routing feature is not turned on [ 600.757291] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 603.055366] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:57:06 (1787396226) [ 604.848945] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.15@tcp is being used as a gateway but routing feature is not turned on [ 604.884114] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 605.063591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg610-server [ 607.163661] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:57:10 (1787396230) [ 607.309982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.349339] LNet: 31511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.364415] LNet: 31511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 607.383300] LNetError: 31511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.405157] LNet: Removed LNI 192.168.206.10@tcp [ 607.435911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.520430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.544164] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 608.240674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[11-16/2]@tcp [ 608.278829] LNet: 31629:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 608.297072] LNet: 31629:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 608.325907] LNetError: 31516:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.11@tcp has gone from up to down [ 608.335349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[11-16/2]@tcp [ 608.337971] LNetError: 31516:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 614.583485] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:57:18 (1787396238) [ 614.876557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.906838] LNetError: 32221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.974729] LNet: Removed LNI 192.168.206.10@tcp [ 616.028822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.166797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 616.244297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.295331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.409546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.490323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.541553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.623575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 616.697459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.734351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.783098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.853748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.888894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.940161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 617.009912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.039913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.107341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.195873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.228868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.314212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 617.388986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.456593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.539026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.640700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.677954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.728249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 617.825162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.858966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.920334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.002881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.041988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.096435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 618.177950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.233343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.338275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 624.610460] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:57:28 (1787396248) [ 626.314951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg610-server [ 628.519537] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:57:32 (1787396252) [ 630.166071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg610-server [ 631.998479] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:57:36 (1787396256) [ 632.907732] Key type .llcrypt unregistered [ 632.913108] Key type ._llcrypt unregistered [ 640.324394] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.460801] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.464819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.468632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 643.174670] Key type ._llcrypt registered [ 643.179928] Key type .llcrypt registered [ 643.244644] LNet: 33902:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 643.244644] mask usage: [+|-] ... [ 643.455891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.497127] alg: No test for adler32 (adler32-zlib) [ 644.390027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.460709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.502173] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 645.204640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 645.232188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.040177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 647.066509] LNet: Removed LNI 10.1.2.3@tcp [ 647.171295] LNet: 34344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.179179] LNetError: 34344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.192545] LNet: Removed LNI 192.168.206.10@tcp [ 647.784492] Key type .llcrypt unregistered [ 647.786304] Key type ._llcrypt unregistered [ 659.287338] Key type ._llcrypt registered [ 659.290201] Key type .llcrypt registered [ 660.811625] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:58:04 (1787396284) [ 661.568255] Key type .llcrypt unregistered [ 661.579948] Key type ._llcrypt unregistered [ 668.186048] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.307323] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.327501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.332111] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.981977] Key type ._llcrypt registered [ 670.984595] Key type .llcrypt registered [ 671.040623] LNet: 35379:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 671.040623] mask usage: [+|-] ... [ 671.194541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.218283] alg: No test for adler32 (adler32-zlib) [ 672.203526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.292740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.331785] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 673.220216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.243568] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.971543] LNet: 35838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.978047] LNetError: 35838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 675.983126] LNet: Removed LNI 10.1.2.3@tcp [ 676.000773] LNet: Removed LNI 192.168.206.10@tcp [ 676.564608] Key type .llcrypt unregistered [ 676.568688] Key type ._llcrypt unregistered [ 688.843736] Key type ._llcrypt registered [ 688.845407] Key type .llcrypt registered [ 690.202808] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:58:33 (1787396313) [ 690.760073] Key type .llcrypt unregistered [ 690.767752] Key type ._llcrypt unregistered [ 696.489269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 696.650227] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 696.654500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 696.660276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.326177] Key type ._llcrypt registered [ 699.329069] Key type .llcrypt registered [ 699.375729] LNet: 36875:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 699.375729] mask usage: [+|-] ... [ 699.531937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.578892] alg: No test for adler32 (adler32-zlib) [ 700.515978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.585643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 700.612338] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 702.248208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 702.265987] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.852854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 703.879635] LNet: Removed LNI 10.1.2.3@tcp [ 703.907908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 703.950720] LNet: Removed LNI 10.1.2.31@tcp [ 703.978774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.000912] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.825501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 704.855085] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 705.800653] LNet: 37758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.806925] LNetError: 37758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 705.814335] LNet: Removed LNI 10.1.2.31@tcp [ 705.833631] LNet: Removed LNI 10.1.2.3@tcp [ 706.475320] Key type .llcrypt unregistered [ 706.482455] Key type ._llcrypt unregistered [ 719.661548] Key type ._llcrypt registered [ 719.663352] Key type .llcrypt registered [ 720.767459] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:59:04 (1787396344) [ 721.165400] LNet: 38551:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 721.165400] mask usage: [+|-] ... [ 721.332467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.354873] alg: No test for adler32 (adler32-zlib) [ 722.335382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.411221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.440189] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 723.093332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.111736] LNet: 38711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.116241] LNetError: 38711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.132790] LNet: Removed LNI 192.168.206.10@tcp [ 723.155417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.225567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.10@tcp [ 723.260801] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 723.986765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.10@tcp [ 724.018996] LNet: Removed LNI 192.168.206.10@tcp [ 724.079895] LNet: 38846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.700661] Key type .llcrypt unregistered [ 724.704552] Key type ._llcrypt unregistered [ 737.176883] Key type ._llcrypt registered [ 737.181798] Key type .llcrypt registered [ 738.386476] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:59:22 (1787396362) [ 740.313144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg610-server [ 741.856431] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:59:26 (1787396366) [ 742.304407] Key type .llcrypt unregistered [ 742.306675] Key type ._llcrypt unregistered [ 748.454760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.658472] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.687333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.692518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.347882] Key type ._llcrypt registered [ 751.351805] Key type .llcrypt registered [ 751.407409] LNet: 40226:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 751.407409] mask usage: [+|-] ... [ 751.609491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.634954] alg: No test for adler32 (adler32-zlib) [ 752.522814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.562763] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 753.228692] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 755.030683] LNet: 40685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.039187] LNetError: 40685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.046224] LNet: Removed LNI 192.168.206.10@tcp1 [ 755.071855] LNet: Removed LNI 10.1.2.3@tcp2 [ 755.626406] Key type .llcrypt unregistered [ 755.629116] Key type ._llcrypt unregistered [ 767.178942] Key type ._llcrypt registered [ 767.181326] Key type .llcrypt registered [ 768.427645] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:59:52 (1787396392) [ 768.730303] LNet: 41489:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 768.730303] mask usage: [+|-] ... [ 768.838437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.852321] alg: No test for adler32 (adler32-zlib) [ 769.817228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.891240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.931118] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 770.824668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.839826] LNet: 41655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.851981] LNetError: 41655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.891353] LNet: Removed LNI 192.168.206.10@tcp [ 770.915603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.010595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 771.025567] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 777.919520] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:00:01 (1787396401) [ 778.062601] LNet: 42242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.077770] LNetError: 42242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.097263] LNet: Removed LNI 192.168.206.10@tcp [ 778.693482] Key type .llcrypt unregistered [ 778.695436] Key type ._llcrypt unregistered [ 784.319887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.456509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.463869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.471975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.045461] Key type ._llcrypt registered [ 787.047551] Key type .llcrypt registered [ 787.079722] LNet: 42523:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 787.079722] mask usage: [+|-] ... [ 787.213924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.237950] alg: No test for adler32 (adler32-zlib) [ 788.184219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.241388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.269455] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 789.060978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 789.096077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.614662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.646902] LNet: 42989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.658609] LNetError: 42989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 791.669027] LNet: Removed LNI 10.1.2.3@tcp [ 791.691564] LNet: Removed LNI 192.168.206.10@tcp [ 791.740908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.845576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 791.891270] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 791.895737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.459763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.498711] LNet: 43016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.511994] LNetError: 43016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.525983] LNetError: 43016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.537900] LNet: Removed LNI 10.1.2.3@tcp [ 792.574720] LNet: Removed LNI 192.168.206.10@tcp [ 792.619926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.699324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 792.739279] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 792.744854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.204651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 793.221407] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 794.306761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.332460] LNet: 43178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.338745] LNetError: 43178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 794.348933] LNetError: 43178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 794.366810] LNet: Removed LNI 10.1.2.3@tcp [ 794.368337] LNet: Removed LNI 192.168.206.10@tcp [ 794.385386] LNet: Removed LNI 192.168.206.10@tcp2 [ 794.422960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.515161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 794.536926] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 794.541274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.549269] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 803.113789] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:00:26 (1787396426) [ 803.376369] LNet: 43774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.389497] LNetError: 43774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 803.408169] LNetError: 43774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 803.422953] LNet: Removed LNI 10.1.2.3@tcp [ 803.432504] LNet: Removed LNI 192.168.206.10@tcp [ 803.447466] LNet: Removed LNI 192.168.206.10@tcp2 [ 804.077331] Key type .llcrypt unregistered [ 804.083162] Key type ._llcrypt unregistered [ 810.239511] Key type ._llcrypt registered [ 810.242401] Key type .llcrypt registered [ 810.293475] LNet: 44001:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 810.293475] mask usage: [+|-] ... [ 810.457396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.489759] alg: No test for adler32 (adler32-zlib) [ 811.441278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.517789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.550723] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 812.392317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.432880] LNet: 44170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.441628] LNetError: 44170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.461115] LNet: Removed LNI 192.168.206.10@tcp [ 812.478440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.530591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 812.550919] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 812.989970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.023321] LNet: 44200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.034882] LNetError: 44200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.055759] LNet: Removed LNI 192.168.206.10@tcp [ 813.082195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.146632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.183884] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 813.748215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.790689] LNet: Removed LNI 192.168.206.10@tcp [ 813.817501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.893333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.924058] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 814.503041] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.734543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.756932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.768455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.215042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.229937] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.240978] LNet: 44309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 817.250671] LNetError: 44309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.259295] LNetError: 44309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.277139] LNet: Removed LNI 192.168.206.10@tcp [ 817.309780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.366777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.385375] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 824.172625] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:00:48 (1787396448) [ 824.586819] LNet: 44906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.607026] LNetError: 44906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.622296] LNet: Removed LNI 192.168.206.10@tcp [ 825.375988] Key type .llcrypt unregistered [ 825.381471] Key type ._llcrypt unregistered [ 832.375958] Key type ._llcrypt registered [ 832.382179] Key type .llcrypt registered [ 832.432585] LNet: 45131:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 832.432585] mask usage: [+|-] ... [ 832.656785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.674263] alg: No test for adler32 (adler32-zlib) [ 833.653710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.750990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.796731] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 835.024095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.078703] LNet: 45304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.087785] LNetError: 45304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.116719] LNet: Removed LNI 192.168.206.10@tcp [ 835.154835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.250992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 835.330128] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 843.758416] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:01:07 (1787396467) [ 844.050812] LNet: 45910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.063574] LNetError: 45910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.076335] LNet: Removed LNI 192.168.206.10@tcp [ 844.916556] Key type .llcrypt unregistered [ 844.920768] Key type ._llcrypt unregistered [ 852.722598] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.777041] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.779536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.782521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.454767] Key type ._llcrypt registered [ 855.456056] Key type .llcrypt registered [ 855.539936] LNet: 46187:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 855.539936] mask usage: [+|-] ... [ 855.746275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.760509] alg: No test for adler32 (adler32-zlib) [ 856.700436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.777227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.828095] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 857.811326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.834856] LNet: 46361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.839152] LNetError: 46361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.849928] LNet: Removed LNI 192.168.206.10@tcp [ 857.867767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.922306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 857.948976] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 866.036184] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:01:29 (1787396489) [ 866.194790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.274697] LNet: 46954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.287294] LNetError: 46954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.320354] LNet: Removed LNI 192.168.206.10@tcp [ 866.351813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.493955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 874.183319] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:01:38 (1787396498) [ 874.299240] LNet: 47540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.021585] Key type .llcrypt unregistered [ 875.024357] Key type ._llcrypt unregistered [ 881.293353] Key type ._llcrypt registered [ 881.303736] Key type .llcrypt registered [ 881.385832] LNet: 47766:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 881.385832] mask usage: [+|-] ... [ 881.644191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.665878] alg: No test for adler32 (adler32-zlib) [ 882.706090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.830699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.864917] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 883.934537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.962930] LNet: 47938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.975342] LNetError: 47938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.018993] LNet: Removed LNI 192.168.206.10@tcp [ 884.069920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.166664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 884.224440] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 891.108914] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:01:55 (1787396515) [ 891.297336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.312187] LNet: 48531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.318987] LNetError: 48531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.342148] LNet: Removed LNI 192.168.206.10@tcp [ 891.358550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.397886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 897.796596] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:02:01 (1787396521) [ 899.574682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 902.051206] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:02:05 (1787396525) [ 904.441656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 906.218355] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:02:10 (1787396530) [ 908.175986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 909.749517] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:02:13 (1787396533) [ 911.977540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 913.939284] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:02:17 (1787396537) [ 915.880258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 917.778415] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:02:21 (1787396541) [ 918.169270] LNet: 50586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.836296] Key type .llcrypt unregistered [ 918.837924] Key type ._llcrypt unregistered [ 925.630193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.763585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.765744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.768565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.330382] Key type ._llcrypt registered [ 928.332443] Key type .llcrypt registered [ 928.366361] LNet: 50868:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 928.366361] mask usage: [+|-] ... [ 928.488827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.505106] alg: No test for adler32 (adler32-zlib) [ 929.490252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.581572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 929.598288] LNet: Added LNI 192.168.206.10@tcp [8/512/0/180] [ 929.603368] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 929.605904] LNet: 50906:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 929.701990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.728851] LNet: 50925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.742453] LNetError: 50925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.749496] LNet: Removed LNI 10.1.2.3@tcp [ 929.750977] LNetError: 50925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.768060] LNet: Removed LNI 192.168.206.10@tcp [ 929.783882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.840519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 929.862737] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 929.936682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.954150] LNet: 50949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.959786] LNetError: 50949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.977811] LNet: Removed LNI 192.168.206.10@tcp [ 930.005402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.082783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 930.106896] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 930.119455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.122564] LNet: 50957:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 930.252171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.274507] LNet: 50975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.284367] LNetError: 50975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 930.295829] LNet: Removed LNI 10.1.2.3@tcp [ 930.297512] LNetError: 50975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.318216] LNet: Removed LNI 192.168.206.10@tcp [ 930.347034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.441907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 930.475240] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 930.480982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.484563] LNet: 50983:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 930.672147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.685276] LNet: 51009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.689733] LNetError: 51009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 930.694143] LNet: Removed LNI 10.1.2.3@tcp [ 930.701333] LNetError: 51009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.715639] LNet: Removed LNI 192.168.206.10@tcp [ 930.728735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.779318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 930.795916] LNet: Added LNI 192.168.206.10@tcp [8/256/0/182] [ 930.799726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 930.896584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.919696] LNet: 51037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.931464] LNetError: 51037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 930.936409] LNet: Removed LNI 10.1.2.3@tcp [ 930.940072] LNetError: 51037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.956628] LNet: Removed LNI 192.168.206.10@tcp [ 930.973668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.026586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.056691] LNet: Added LNI 192.168.206.10@tcp [8/256/0/182] [ 931.059724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 931.062441] LNet: 51045:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 931.150604] LNet: Removed LNI 10.1.2.3@tcp [ 931.169415] LNet: Removed LNI 192.168.206.10@tcp [ 931.866332] Key type .llcrypt unregistered [ 931.868652] Key type ._llcrypt unregistered [ 942.614788] Key type ._llcrypt registered [ 942.616674] Key type .llcrypt registered [ 943.903780] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:02:47 (1787396567) [ 944.301717] LNet: 51859:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 944.301717] mask usage: [+|-] ... [ 944.497083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.508052] alg: No test for adler32 (adler32-zlib) [ 945.397652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.443510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.465301] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 946.182137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.10 [ 946.210616] LNet: 52020:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 946.218105] LNet: Removed LNI 192.168.206.10@tcp [ 952.167307] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:02:56 (1787396576) [ 952.364264] LNet: 52589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.379693] LNetError: 52589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.417833] LNet: Removed LNI 192.168.206.10@tcp [ 953.129304] Key type .llcrypt unregistered [ 953.133072] Key type ._llcrypt unregistered [ 959.148765] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 959.371959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 962.036964] Key type ._llcrypt registered [ 962.045429] Key type .llcrypt registered [ 962.078982] LNet: 52872:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 962.078982] mask usage: [+|-] ... [ 962.206559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 962.231786] alg: No test for adler32 (adler32-zlib) [ 963.620582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.953158] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:03:13 (1787396593) [ 970.304733] LNet: 53503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.309717] LNetError: 53503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.320908] LNet: Removed LNI 10.1.2.3@tcp [ 970.964273] Key type .llcrypt unregistered [ 970.966144] Key type ._llcrypt unregistered [ 977.699070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 977.993809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.001122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.015300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.699398] Key type ._llcrypt registered [ 979.702433] Key type .llcrypt registered [ 979.766246] LNet: 53768:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 979.766246] mask usage: [+|-] ... [ 979.961404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.979169] alg: No test for adler32 (adler32-zlib) [ 980.906990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 986.742416] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:03:30 (1787396610) [ 988.075604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 989.474344] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:03:33 (1787396613) [ 989.678424] LNet: 54679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.683870] LNetError: 54679:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.700849] LNet: Removed LNI 10.1.2.3@tcp [ 990.173776] Key type .llcrypt unregistered [ 990.177574] Key type ._llcrypt unregistered [ 996.309478] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 996.544685] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 996.555822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 996.565991] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 999.204424] Key type ._llcrypt registered [ 999.207268] Key type .llcrypt registered [ 999.243425] LNet: 54951:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 999.243425] mask usage: [+|-] ... [ 999.379230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.389321] alg: No test for adler32 (adler32-zlib) [ 1000.371391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.488829] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.903680] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:03:52 (1787396632) [ 1009.513264] LNet: 55906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.518588] LNetError: 55906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.534780] LNet: Removed LNI 10.1.2.3@tcp [ 1010.096530] Key type .llcrypt unregistered [ 1010.098126] Key type ._llcrypt unregistered [ 1016.571789] Key type ._llcrypt registered [ 1016.573882] Key type .llcrypt registered [ 1016.636200] LNet: 56130:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1016.636200] mask usage: [+|-] ... [ 1016.852735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.863324] alg: No test for adler32 (adler32-zlib) [ 1017.826463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1017.868350] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1025.807612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1026.584853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1026.650873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1026.738204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1026.770435] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1027.068638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.103650] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1027.105506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1027.137919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.166405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.210291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.255711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.302481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.349049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.366872] LNet: 56344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.391735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.426294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.454393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.497718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.543838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.596541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.631848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1028.170292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1028.229142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1028.277792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1028.306827] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1028.317141] LNet: Skipped 1 previous similar message [ 1028.522646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.549848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.580178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.605407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.638121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.693756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.721645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.744936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.759685] LNet: 56453:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.777644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.811239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.867149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.904212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.953359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.982816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.019634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.606262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1029.647465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1029.693530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1029.725503] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1029.957706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.994650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.037075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.109194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.168568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.229649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.286842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.343515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.372464] LNet: 56562:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.395579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.454782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.512472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.577841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.643389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.699930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.764557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.250363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1031.296244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1031.347411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1031.603451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.656990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1031.691096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.720134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.757444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.801500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.858940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.904590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.938392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.971134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.005820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.037327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.083221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.115461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.185313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.842958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1032.901815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1032.976120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1032.994497] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1033.003563] LNet: Skipped 1 previous similar message [ 1033.232487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.287862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.323739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.360836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.386382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.421613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.455671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.484252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.498880] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.507216] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1033.522504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.552549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.592639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.631778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.665856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.701899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.732682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.369040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1034.420907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1034.457992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1034.736901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.770455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.791783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.818618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.844305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.887124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.920514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.965906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.028460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.061064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.110554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.173384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.217737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.267253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.314499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1035.599609] LNet: 56931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.615425] LNetError: 56931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.638777] LNet: Removed LNI 192.168.206.10@tcp [ 1036.022381] Key type .llcrypt unregistered [ 1036.026371] Key type ._llcrypt unregistered [ 1053.574480] Key type ._llcrypt registered [ 1053.578398] Key type .llcrypt registered [ 1055.102966] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:04:38 (1787396678) [ 1056.017661] Key type .llcrypt unregistered [ 1056.023668] Key type ._llcrypt unregistered [ 1062.748366] Key type ._llcrypt registered [ 1062.753305] Key type .llcrypt registered [ 1062.797225] LNet: 57971:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1062.797225] mask usage: [+|-] ... [ 1063.054451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.068453] alg: No test for adler32 (adler32-zlib) [ 1063.996677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1064.020579] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1071.445516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1081.711784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.777928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1081.808430] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1083.086943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1083.133533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 1 [ 1083.180838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1083.234886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 1 [ 1083.295248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_interrupt -r 1 [ 1083.333477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 1 [ 1083.390161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_interrupt -r 1 [ 1083.436468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 1 [ 1083.470795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1083.493363] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1083.498705] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1083.729598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.773955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1083.801244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.825824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.864953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.903874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.959869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.002836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.026926] LNet: 58400:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.068212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.127772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.181969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.234672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.286705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.343488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.386929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.087529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1085.130947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 1 [ 1085.188889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1085.236753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 1 [ 1085.284374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_dropped -r 1 [ 1085.328856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 1 [ 1085.391625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_dropped -r 1 [ 1085.441122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 1 [ 1085.495374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1085.513919] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1085.521715] LNet: Skipped 2 previous similar messages [ 1085.739922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.773297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.816126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.859728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.913117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.954761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.991673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.061562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.116875] LNet: 58522:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.143637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.205966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.281953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.340604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.397777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.456845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.561073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1087.196178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1087.250941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 1 [ 1087.289228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1087.345784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 1 [ 1087.403370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_aborted -r 1 [ 1087.458413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 1 [ 1087.520670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_aborted -r 1 [ 1087.571788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 1 [ 1087.627990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1087.664811] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1087.671560] LNet: Skipped 2 previous similar messages [ 1087.906862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.936945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1087.972381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.008198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.037382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.070750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.097524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.128629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.142144] LNet: 58643:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1088.158453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.204625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.227780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.260459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.299166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.334935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.366940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.814701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1088.844500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 1 [ 1088.877146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1088.909531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 1 [ 1088.941550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_no_route -r 1 [ 1088.967756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 1 [ 1088.992867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_no_route -r 1 [ 1089.019560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 1 [ 1089.055988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1089.254947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.280914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1089.316421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.356434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.404161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.438472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.471430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.535307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.590926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.622922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.662029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.714042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.765240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.794270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.828898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.369658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1090.444794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 1 [ 1090.498075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1090.581584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 1 [ 1090.643771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_timeout -r 1 [ 1090.706933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 1 [ 1090.757594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_timeout -r 1 [ 1090.820789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 1 [ 1090.902264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1090.960475] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1090.970498] LNet: Skipped 5 previous similar messages [ 1091.260166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.302525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1091.342569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.388451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.426802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.483674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.529457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.589287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.613292] LNet: 58885:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.621043] LNet: 58885:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1091.640155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.692551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.743864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.796312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.846825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.904142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.963246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.567754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1092.619603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_error -r 1 [ 1092.658533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1092.692850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_error -r 1 [ 1092.723475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_error -r 1 [ 1092.752660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_error -r 1 [ 1092.783298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_error -r 1 [ 1092.828949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_error -r 1 [ 1092.873319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1093.149481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.188579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1093.225460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.270536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.307851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.330277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.360327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.384780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.414490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.447852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.483670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.507327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.535564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.569879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.604839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1099.993696] LNet: 59082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.999310] LNetError: 59082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.004810] LNet: Removed LNI 192.168.206.10@tcp [ 1101.064125] LNet: Removed LNI 192.168.206.10@tcp1 [ 1101.596574] Key type .llcrypt unregistered [ 1101.599053] Key type ._llcrypt unregistered [ 1118.769296] Key type ._llcrypt registered [ 1118.773455] Key type .llcrypt registered [ 1120.287351] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:05:43 (1787396743) [ 1121.373375] Key type .llcrypt unregistered [ 1121.380873] Key type ._llcrypt unregistered [ 1127.870915] Key type ._llcrypt registered [ 1127.872372] Key type .llcrypt registered [ 1127.934899] LNet: 60118:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1127.934899] mask usage: [+|-] ... [ 1128.140408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.157328] alg: No test for adler32 (adler32-zlib) [ 1129.106269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1129.131228] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1136.867883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1137.491675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1137.540762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1137.579925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1137.609200] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1137.828764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.860897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1137.886238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.914852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.945533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.990134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.039866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.083454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.116158] LNet: 60332:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.139057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.203792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.246993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.290648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.339737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.396297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.478029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.956691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1138.999415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1139.029670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1139.045440] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1139.250904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.275372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1139.300937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.322380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.352659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.395398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.447095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.493700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.524543] LNet: 60442:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.555911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.604533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.658290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.695254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.770618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.830219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.871643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.406791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1140.454167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1140.489950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1140.519763] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1140.732982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.769235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.799201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.823858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.847880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.893241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.924501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.955480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.975338] LNet: 60551:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.996172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.021887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.071162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.101041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.132708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.171748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.191072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.451227] LNet: 60592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.458596] LNetError: 60592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.483840] LNet: Removed LNI 192.168.206.10@tcp [ 1141.750636] Key type .llcrypt unregistered [ 1141.753375] Key type ._llcrypt unregistered [ 1161.395120] Key type ._llcrypt registered [ 1161.399471] Key type .llcrypt registered [ 1162.952833] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:06:26 (1787396786) [ 1164.399384] Key type .llcrypt unregistered [ 1164.401498] Key type ._llcrypt unregistered [ 1171.144971] Key type ._llcrypt registered [ 1171.146305] Key type .llcrypt registered [ 1171.171781] LNet: 61632:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1171.171781] mask usage: [+|-] ... [ 1171.287781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.305313] alg: No test for adler32 (adler32-zlib) [ 1172.358070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.438076] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1180.770939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1190.155852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.209873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1190.248240] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1191.655223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1191.689485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 1 [ 1191.718762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1191.772994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 1 [ 1191.854884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_dropped -r 1 [ 1191.899396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 1 [ 1191.946498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_dropped -r 1 [ 1192.012786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 1 [ 1192.062329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1192.099779] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1192.113773] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1192.364664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.394558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.430595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.458579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.508941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.550467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.610559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.646057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.666499] LNet: 62062:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.696358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.756761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.786805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.822942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.858654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.907917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.961730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1192.994248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.600094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1193.659397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_error -r 1 [ 1193.707600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1193.787551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_error -r 1 [ 1193.880097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_error -r 1 [ 1193.917969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_error -r 1 [ 1193.975925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_error -r 1 [ 1194.021275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_error -r 1 [ 1194.062608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1194.083701] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1194.087832] LNet: Skipped 2 previous similar messages [ 1194.337228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.381251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1194.408453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.437406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.462827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.492217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.519888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.544792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.558703] LNet: 62185:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.569797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.591544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.612370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.635212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.654331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.674891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.697188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.722590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.158874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1195.202044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_timeout -r 1 [ 1195.253544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1195.302797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_timeout -r 1 [ 1195.337632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_timeout -r 1 [ 1195.375350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_timeout -r 1 [ 1195.415744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_timeout -r 1 [ 1195.470316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_timeout -r 1 [ 1195.506380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.110@tcp [ 1195.538616] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1195.763906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.798769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1195.840347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.869522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.902658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.931565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.971693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.017404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.049510] LNet: 62309:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.070530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.121892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.177496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.247835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.301179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.372790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.428470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.451686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.316779] LNet: 62387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.324482] LNetError: 62387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.333600] LNet: Removed LNI 192.168.206.10@tcp [ 1201.346872] LNet: Removed LNI 192.168.206.10@tcp1 [ 1201.695528] Key type .llcrypt unregistered [ 1201.698502] Key type ._llcrypt unregistered [ 1216.486018] Key type ._llcrypt registered [ 1216.488747] Key type .llcrypt registered [ 1217.710585] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:07:21 (1787396841) [ 1219.168577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1220.670652] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:07:24 (1787396844) [ 1221.348405] Key type .llcrypt unregistered [ 1221.353199] Key type ._llcrypt unregistered [ 1227.183564] Key type ._llcrypt registered [ 1227.186770] Key type .llcrypt registered [ 1227.219213] LNet: 63716:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1227.219213] mask usage: [+|-] ... [ 1227.357915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.370504] alg: No test for adler32 (adler32-zlib) [ 1228.293825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.315575] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1235.953454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1236.783978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1236.826624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1236.878519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1236.903747] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1237.180096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.220115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1237.258951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.295453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.349944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.409631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.458496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.492610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.537891] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1237.542676] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1237.549269] LNet: 63931:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.577628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.621467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.669317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.709826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.744915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.774960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.056270] LNet: 63970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.084772] LNetError: 63970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.107574] LNet: Removed LNI 192.168.206.10@tcp [ 1238.574411] Key type .llcrypt unregistered [ 1238.577518] Key type ._llcrypt unregistered [ 1257.400393] Key type ._llcrypt registered [ 1257.402321] Key type .llcrypt registered [ 1257.454112] LNet: 64421:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1257.454112] mask usage: [+|-] ... [ 1257.668820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1257.685797] alg: No test for adler32 (adler32-zlib) [ 1258.600287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1258.631544] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1266.145607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1274.887929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.948020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1274.975675] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1276.975508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1277.017413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e network_timeout -r 1 [ 1277.075103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1277.144374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e network_timeout -r 1 [ 1277.191406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e network_timeout -r 1 [ 1277.263897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e network_timeout -r 1 [ 1277.322244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e network_timeout -r 1 [ 1277.371372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e network_timeout -r 1 [ 1277.426159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1277.460374] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1277.711204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.750430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1277.798676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.831496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.868928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.916697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.952604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.987898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.000704] LNet: 64851:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.017261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.054063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.074570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.103376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.117621] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1278.120118] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1278.128609] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1278.132152] LNet: Skipped 1 previous similar message [ 1278.145556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.189351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.684863] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1284.166605] LNet: 64925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.191873] LNetError: 64925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.216285] LNet: Removed LNI 192.168.206.10@tcp [ 1284.243173] LNet: Removed LNI 192.168.206.10@tcp1 [ 1284.714403] Key type .llcrypt unregistered [ 1284.715956] Key type ._llcrypt unregistered [ 1301.576790] Key type ._llcrypt registered [ 1301.584552] Key type .llcrypt registered [ 1303.060973] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:08:46 (1787396926) [ 1303.324534] LNet: 65725:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1303.324534] mask usage: [+|-] ... [ 1303.549935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.576423] alg: No test for adler32 (adler32-zlib) [ 1304.508359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.583325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.613278] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1305.599399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.644140] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1306.632296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1306.726396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1306.791474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1306.834832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1306.878160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1306.920469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1306.958369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1306.976710] LNetError: 66025:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1307.615940] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1307.619267] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1309.666615] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1310.607924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1313.763558] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1314.123248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1314.157627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1314.211242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.227224] LNet: 66133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.234904] LNetError: 66133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.243691] LNet: Removed LNI 192.168.206.10@tcp [ 1314.253507] LNet: Removed LNI 192.168.206.10@tcp1 [ 1314.284073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.329607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1314.353599] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1315.116679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.140437] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1316.161760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1316.216314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.293733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1316.351047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1316.411353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1316.470218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1316.510579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1316.545855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1316.571836] LNetError: 66405:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1317.411692] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1320.176571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1323.553248] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1323.562209] LNet: Skipped 1 previous similar message [ 1328.265772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1328.329102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1328.382688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.578455] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:09:19 (1787396959) [ 1335.641085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.661339] LNet: 67116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.668596] LNetError: 67116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.674305] LNetError: 67116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1335.686724] LNet: Removed LNI 192.168.206.10@tcp [ 1335.716506] LNet: Removed LNI 192.168.206.10@tcp1 [ 1335.745294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.783790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1335.798623] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1336.473376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.491201] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1337.415255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1337.475812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1337.509426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1337.556890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1337.589375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1337.621585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1337.651849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 0 [ 1337.757919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1337.824418] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.10@tcp [ 1338.861693] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1338.877132] LNet: Skipped 1 previous similar message [ 1352.474650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1352.659076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1352.699738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.751489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 500 [ 1352.897386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.597933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.650896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.691307] LNet: 67598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.702517] LNetError: 67598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.709542] LNetError: 67598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1358.722113] LNet: Removed LNI 192.168.206.10@tcp [ 1358.744417] LNet: Removed LNI 192.168.206.10@tcp1 [ 1358.775615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.877509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1358.918939] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1359.752657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1359.795644] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1360.642625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1360.679372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.784029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1360.834768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1360.903970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1360.945409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1360.980605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1361.003199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.10@tcp --health 0 [ 1362.911368] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1362.916113] LNet: Skipped 3 previous similar messages [ 1363.466304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.793619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.851236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1372.912910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.493996] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:10:03 (1787397003) [ 1379.900798] LNet: 68645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.909087] LNetError: 68645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.925128] LNetError: 68645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1379.932152] LNet: Removed LNI 192.168.206.10@tcp [ 1379.944569] LNet: Removed LNI 192.168.206.10@tcp1 [ 1380.612419] Key type .llcrypt unregistered [ 1380.628952] Key type ._llcrypt unregistered [ 1386.291227] Key type ._llcrypt registered [ 1386.292748] Key type .llcrypt registered [ 1386.343733] LNet: 68868:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1386.343733] mask usage: [+|-] ... [ 1386.541759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.558343] alg: No test for adler32 (adler32-zlib) [ 1387.529797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1387.562709] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1396.141776] Lustre: DEBUG MARKER: Initial discovery [ 1396.210830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1398.445631] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1398.696393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1398.734319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1398.756183] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1398.790069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1398.816985] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1398.818813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.110@tcp -r 1 -m GET -l 3 [ 1398.876793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.110@tcp -r 1 -m GET -e local_error [ 1398.912904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.110@tcp -r 1 -m PUT -l 6 [ 1400.118898] Lustre: DEBUG MARKER: Force oleg610-client.virtnet to discover 192.168.206.110@tcp (in background) [ 1400.318779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.110@tcp [ 1401.581514] Lustre: DEBUG MARKER: Force oleg610-server to discover 192.168.206.10@tcp [ 1403.378166] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1404.283460] Lustre: DEBUG MARKER: Wait for 69408 [ 1405.539898] Lustre: DEBUG MARKER: Finished wait on 69408 [ 1405.671921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1405.714637] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1405.731922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1405.949458] LNet: 69859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.964826] LNetError: 69859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.977958] LNet: Removed LNI 192.168.206.10@tcp [ 1406.342452] Key type .llcrypt unregistered [ 1406.344934] Key type ._llcrypt unregistered [ 1422.197159] Key type ._llcrypt registered [ 1422.200676] Key type .llcrypt registered [ 1423.425670] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:10:47 (1787397047) [ 1424.030987] Key type .llcrypt unregistered [ 1424.033907] Key type ._llcrypt unregistered [ 1429.598121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1429.755013] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1429.760505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1429.763688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1431.327262] Key type ._llcrypt registered [ 1431.330342] Key type .llcrypt registered [ 1431.381399] LNet: 70894:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1431.381399] mask usage: [+|-] ... [ 1431.545354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.554588] alg: No test for adler32 (adler32-zlib) [ 1432.480610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.546316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.567942] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1433.242915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1433.266897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1441.653868] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:11:05 (1787397065) [ 1441.787865] LNet: 71921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.794995] LNetError: 71921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1441.800883] LNet: Removed LNI 10.1.2.3@tcp [ 1441.821092] LNet: Removed LNI 192.168.206.10@tcp [ 1442.658960] Key type .llcrypt unregistered [ 1442.662558] Key type ._llcrypt unregistered [ 1449.680785] Key type ._llcrypt registered [ 1449.687476] Key type .llcrypt registered [ 1449.716349] LNet: 72177:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1449.716349] mask usage: [+|-] ... [ 1449.812441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1449.823239] alg: No test for adler32 (adler32-zlib) [ 1450.831532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.897529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.953721] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1451.774247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1451.786222] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1452.873699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.10@tcp [ 1453.922683] LNetError: Unexpected error -22 connecting to 192.168.206.10@tcp at host 192.168.206.10:988 [ 1461.192820] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:11:25 (1787397085) [ 1461.677213] LNet: 73060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.693773] LNetError: 73060:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1461.702845] LNet: Removed LNI 10.1.2.3@tcp [ 1461.714031] LNet: Removed LNI 192.168.206.10@tcp [ 1462.694339] Key type .llcrypt unregistered [ 1462.698487] Key type ._llcrypt unregistered [ 1469.793989] Key type ._llcrypt registered [ 1469.795518] Key type .llcrypt registered [ 1469.838884] LNet: 73292:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1469.838884] mask usage: [+|-] ... [ 1470.072280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.104355] alg: No test for adler32 (adler32-zlib) [ 1471.109565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.196035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.232791] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1471.942902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1471.975760] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 1473.123296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.10@tcp1 --nid 192.168.206.10@tcp2 [ 1477.704918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp1 192.168.206.10@tcp2 [ 1477.724179] LNetError: 73897:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.10@tcp1 to 192.168.206.10@tcp2 - Route table may be misconfigured [ 1477.748101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp2 192.168.206.10@tcp1 [ 1484.209535] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:11:48 (1787397108) [ 1484.322412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.353397] LNet: 74466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.362423] LNetError: 74466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.370058] LNet: Removed LNI 192.168.206.10@tcp1 [ 1485.415540] LNet: Removed LNI 192.168.206.10@tcp2 [ 1485.432546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.491154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.531257] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1486.112986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.128706] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1486.791576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 1486.832466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.10@tcp [ 1486.889111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp1 192.168.206.10@tcp1 [ 1486.940173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.10@tcp -d 192.168.206.10@tcp -e network_timeout [ 1487.001749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -e network_timeout [ 1487.051496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -e network_timeout [ 1487.110661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -e network_timeout [ 1487.176627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1487.207543] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1487.235756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.347959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1487.454285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1487.520502] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1493.216313] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:11:57 (1787397117) [ 1493.280827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.298327] LNet: 75324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.303527] LNetError: 75324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.311101] LNetError: 75324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1493.322050] LNet: Removed LNI 192.168.206.10@tcp [ 1494.381607] LNet: Removed LNI 192.168.206.10@tcp1 [ 1494.419425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.481188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1494.689272] LNet: 75370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.067383] Key type .llcrypt unregistered [ 1495.070502] Key type ._llcrypt unregistered [ 1505.401659] Key type ._llcrypt registered [ 1505.406370] Key type .llcrypt registered [ 1506.471634] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:12:10 (1787397130) [ 1507.110524] LNet: 76141:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1507.110524] mask usage: [+|-] ... [ 1507.352649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.387181] alg: No test for adler32 (adler32-zlib) [ 1508.389640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.121564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1511.888393] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:12:15 (1787397135) [ 1512.265718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.299610] LNet: 76468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.329092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.399152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.426765] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1513.174679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1513.201279] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1514.744433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1514.778953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp1 [ 1514.820475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp1 [ 1521.520178] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:12:25 (1787397145) [ 1523.557813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg610-server [ 1526.290439] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:12:29 (1787397149) [ 1528.759077] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg610-server [ 1530.986395] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:12:34 (1787397154) [ 1532.715327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg610-server [ 1534.833592] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:12:38 (1787397158) [ 1537.470811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg610-server [ 1539.535388] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:12:43 (1787397163) [ 1542.482463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg610-server [ 1545.669669] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:12:48 (1787397168) [ 1547.831902] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg610-server [ 1550.072303] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:12:53 (1787397173) [ 1552.276436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg610-server [ 1554.331712] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:12:57 (1787397177) [ 1556.182109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg610-server [ 1558.113801] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:13:01 (1787397181) [ 1560.211299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg610-server [ 1561.904708] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:13:05 (1787397185) [ 1562.291756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.333469] LNet: 80349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.339247] LNetError: 80349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.344848] LNet: Removed LNI 192.168.206.10@tcp [ 1563.366385] LNet: Removed LNI 192.168.206.10@tcp1 [ 1563.424757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.515318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.547851] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1564.606608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1564.714756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1564.922447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.954426] LNet: 80496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.965861] LNetError: 80496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.975171] LNetError: 80496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1564.997291] LNet: Removed LNI 192.168.206.10@tcp [ 1565.019752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.089607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1565.117158] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1565.956672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1566.167579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.213824] LNetError: 80641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.274700] LNet: Removed LNI 192.168.206.10@tcp [ 1567.294962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.357807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.394813] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1568.395707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1568.541736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1569.934422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.969449] LNet: 80798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.975374] LNet: 80798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1569.990826] LNetError: 80798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.013813] LNet: Removed LNI 192.168.206.10@tcp [ 1570.076142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.146685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1570.175735] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1571.074455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1571.318061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.400695] LNet: Removed LNI 192.168.206.10@tcp [ 1572.468455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.512992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.534659] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1573.367257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1573.469530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1573.752544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.822736] LNet: Removed LNI 192.168.206.10@tcp [ 1574.880209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.983626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1575.019371] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1575.774306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1575.981982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.016648] LNet: 81236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.021467] LNet: 81236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1576.027746] LNetError: 81236:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.033641] LNetError: 81236:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1576.051251] LNet: Removed LNI 192.168.206.10@tcp [ 1576.084611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.166646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.193245] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1577.048672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1577.137212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1577.355857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.414441] LNet: Removed LNI 192.168.206.10@tcp [ 1578.485891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.584921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1578.639114] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1579.440708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1579.693942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.773609] LNet: Removed LNI 192.168.206.10@tcp [ 1580.801993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.893202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.931701] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1581.747524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1581.818976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1582.036081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.076917] LNet: Removed LNI 192.168.206.10@tcp [ 1583.136231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.195793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1583.249284] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1584.333736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1584.619778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.652890] LNet: 81822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.662683] LNet: 81822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1584.678034] LNetError: 81822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.687170] LNetError: 81822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1584.701702] LNet: Removed LNI 192.168.206.10@tcp [ 1584.745656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.804540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.842450] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1585.550388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1585.656605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1585.944834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.051034] LNet: Removed LNI 192.168.206.10@tcp [ 1587.092499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.193631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1587.245202] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1588.498672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1588.688927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.740831] LNet: Removed LNI 192.168.206.10@tcp [ 1589.784401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.875091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.907578] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1590.695427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1590.785761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.824766] LNet: Removed LNI 192.168.206.10@tcp [ 1590.850258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.900063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.941060] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1591.861844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1599.231748] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:13:43 (1787397223) [ 1599.483978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.508330] LNet: Removed LNI 192.168.206.10@tcp [ 1599.529633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.574339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1599.601202] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1601.289571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.334570] LNet: 83090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.339914] LNet: 83090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1601.347385] LNetError: 83090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.353400] LNetError: 83090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1601.394360] LNet: Removed LNI 192.168.206.10@tcp231 [ 1601.416690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.541394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1601.581851] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1601.591542] LNet: 83098:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1602.358849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1602.415435] LNet: Removed LNI 192.168.206.10@tcp231 [ 1602.434833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1602.468442] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1603.847716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.884365] LNet: Removed LNI 192.168.206.10@tcp231 [ 1603.919168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.046285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1604.096900] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/0] [ 1604.105247] LNet: 83250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1604.588323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1604.622482] LNet: Removed LNI 192.168.206.10@tcp231 [ 1604.663169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1604.703589] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/0] [ 1605.772241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.805776] LNet: Removed LNI 192.168.206.10@tcp231 [ 1605.837883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.942930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1605.973291] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/60] [ 1605.977966] LNet: 83402:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1606.375674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1606.404569] LNet: Removed LNI 192.168.206.10@tcp231 [ 1606.430682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1606.472099] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/60] [ 1607.822446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.873520] LNet: Removed LNI 192.168.206.10@tcp231 [ 1607.915071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.010538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1608.037945] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1608.044911] LNet: 83553:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.591993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1608.629751] LNet: Removed LNI 192.168.206.10@tcp231 [ 1608.669591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1608.708663] LNet: Added LNI 192.168.206.10@tcp231 [8/256/0/180] [ 1617.186501] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:14:01 (1787397241) [ 1617.540379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.577824] LNet: Removed LNI 192.168.206.10@tcp231 [ 1617.603137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.737742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1617.767308] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1618.692377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1618.756298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1618.792445] LNet: Removed LNI 192.168.206.10@tcp232 [ 1618.814560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1618.837947] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1619.768587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1619.799310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1619.827986] LNet: Removed LNI 192.168.206.10@tcp232 [ 1619.845548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1619.882355] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1620.667198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1620.729564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1620.782050] LNet: Removed LNI 192.168.206.10@tcp232 [ 1620.806863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1620.840189] LNet: Added LNI 192.168.206.10@tcp232 [8/256/0/180] [ 1621.747847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1621.779228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1621.811884] LNet: Removed LNI 192.168.206.10@tcp232 [ 1621.829727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1629.343383] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:14:12 (1787397252) [ 1630.903302] Key type .llcrypt unregistered [ 1630.904929] Key type ._llcrypt unregistered [ 1638.731734] Key type ._llcrypt registered [ 1638.736326] Key type .llcrypt registered [ 1638.800941] LNet: 85622:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1638.800941] mask usage: [+|-] ... [ 1639.196975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.220918] alg: No test for adler32 (adler32-zlib) [ 1640.361692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1640.407625] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1647.995099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1657.690576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.731640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1657.763441] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1659.447211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_interrupt -r 2 [ 1659.488695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 2 [ 1659.534889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_interrupt -r 2 [ 1659.574614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 2 [ 1659.636805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_interrupt -r 2 [ 1659.692584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_interrupt -r 2 [ 1659.752180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_interrupt -r 2 [ 1659.801927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_interrupt -r 2 [ 1659.842321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1659.872909] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1659.883033] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1660.116596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1660.163886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1660.196285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1660.225865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1660.255631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1660.286330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1660.310772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1660.348986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1660.362662] LNet: 86056:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.378687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1660.406668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1660.445328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1660.487590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1660.536747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.574802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.199106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_dropped -r 2 [ 1661.242212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 2 [ 1661.274398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_dropped -r 2 [ 1661.323974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 2 [ 1661.354526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_dropped -r 2 [ 1661.396628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_dropped -r 2 [ 1661.424853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_dropped -r 2 [ 1661.463277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_dropped -r 2 [ 1661.517933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1661.552627] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1661.801195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.841395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1661.899814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.960399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1662.020803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1662.068795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1662.112381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1662.164931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1662.186833] LNet: 86177:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1662.208489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1662.284541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1662.347370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1662.423618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.459168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1662.493598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.142385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_aborted -r 2 [ 1663.193433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 2 [ 1663.229228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_aborted -r 2 [ 1663.267312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 2 [ 1663.324718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_aborted -r 2 [ 1663.366889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_aborted -r 2 [ 1663.400443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_aborted -r 2 [ 1663.434939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_aborted -r 2 [ 1663.468222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1663.517637] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1663.829206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1663.884965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1663.950691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1664.027646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1664.060480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1664.090735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1664.125973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1664.154682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1664.178417] LNet: 86297:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1664.201565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1664.258579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1664.318224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1664.382284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1664.449229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1664.508350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1665.235875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_no_route -r 2 [ 1665.281161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 2 [ 1665.361143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_no_route -r 2 [ 1665.422348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 2 [ 1665.484757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_no_route -r 2 [ 1665.532287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_no_route -r 2 [ 1665.587150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_no_route -r 2 [ 1665.656334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_no_route -r 2 [ 1665.704383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1665.732829] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1666.061553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1666.105412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1666.133869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1666.178756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1666.241856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1666.285306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1666.332117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1666.383531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1666.408975] LNet: 86417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.448126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1666.498707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1666.565291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.610637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.676728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.731767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1667.345556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e local_timeout -r 2 [ 1667.397942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 2 [ 1667.449504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e local_timeout -r 2 [ 1667.511259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 2 [ 1667.564726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e local_timeout -r 2 [ 1667.599919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e local_timeout -r 2 [ 1667.654201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e local_timeout -r 2 [ 1667.730545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e local_timeout -r 2 [ 1667.777678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1668.080050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1668.116220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1668.171603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1668.225090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.285373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1668.317428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1668.352100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.407117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1668.465421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1668.506591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1668.557753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1668.607617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1668.650268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1668.700204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1669.368604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp -m GET -e remote_dropped -r 2 [ 1669.403573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 2 [ 1669.434976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp -m GET -e remote_dropped -r 2 [ 1669.490631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 2 [ 1669.547986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp -m GET -e remote_dropped -r 2 [ 1669.617804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.110@tcp1 -m GET -e remote_dropped -r 2 [ 1669.678572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp -m GET -e remote_dropped -r 2 [ 1669.753644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.10@tcp1 -d 192.168.206.10@tcp1 -m GET -e remote_dropped -r 2 [ 1669.822588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.10@tcp 192.168.206.110@tcp [ 1669.875143] LNet: There was an unexpected network error while writing to 192.168.206.110: rc = -22 [ 1669.891041] LNet: Skipped 1 previous similar message [ 1669.896856] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 1670.141901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.191643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1670.223995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.260813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.293539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1670.329592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1670.403199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1670.467080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1670.492056] LNet: 86657:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1670.505754] LNet: 86657:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1670.520673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1670.550336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1670.589824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1670.630635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1670.669698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1670.714215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1675.914206] LNet: 86732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.922068] LNetError: 86732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.945843] LNet: Removed LNI 192.168.206.10@tcp [ 1675.958705] LNet: Removed LNI 192.168.206.10@tcp1 [ 1676.328439] Key type .llcrypt unregistered [ 1676.333858] Key type ._llcrypt unregistered [ 1694.050377] Key type ._llcrypt registered [ 1694.052541] Key type .llcrypt registered [ 1695.395761] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:15:19 (1787397319) [ 1695.708824] LNet: 87539:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1695.708824] mask usage: [+|-] ... [ 1695.857802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1695.889605] alg: No test for adler32 (adler32-zlib) [ 1696.839932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.893474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.916603] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1697.726823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1697.790237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.10@tcp [ 1697.830521] LNetError: 87713:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1697.889065] LNet: 1 local NIs in recovery (showing 1): 192.168.206.10@tcp [ 1697.933804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.10@tcp [ 1698.041524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1704.770616] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:15:28 (1787397328) [ 1705.003682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.048665] LNet: 88301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.080640] LNetError: 88301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.110656] LNet: Removed LNI 192.168.206.10@tcp [ 1705.135211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.197518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.220370] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1706.133153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1706.148094] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 1707.312719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.10@tcp [ 1707.332121] LNetError: 88568:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.206.10@tcp to recovery [ 1708.259942] LNet: There was an unexpected network error while writing to 192.168.206.10: rc = -22 [ 1708.617371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.10@tcp [ 1717.907833] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:15:41 (1787397341) [ 1719.595521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg610-server [ 1721.656883] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:15:45 (1787397345) [ 1724.021581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg610-server [ 1725.609373] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:15:49 (1787397349) [ 1727.854691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg610-server [ 1729.762447] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:15:53 (1787397353) [ 1729.922397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.977155] LNet: 90201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.987776] LNetError: 90201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.993343] LNet: Removed LNI 192.168.206.10@tcp [ 1730.005302] LNet: Removed LNI 192.168.206.10@tcp1 [ 1730.024330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.073895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.091879] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1730.853433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1730.918345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1730.959448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1731.010476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1731.060759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1731.096640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1738.121989] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:16:02 (1787397362) [ 1738.395985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.425090] LNet: 90926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.443276] LNetError: 90926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.453039] LNetError: 90926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1738.477242] LNet: Removed LNI 192.168.206.10@tcp [ 1738.502189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.606356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.635298] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1746.231256] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:16:10 (1787397370) [ 1746.402266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.427660] LNet: 91629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.435926] LNetError: 91629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.452555] LNet: Removed LNI 192.168.206.10@tcp [ 1746.465996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.525244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.563492] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1753.301432] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:16:17 (1787397377) [ 1755.012945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1756.654460] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:16:20 (1787397380) [ 1757.351398] LNet: 92664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.357609] LNetError: 92664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.375438] LNet: Removed LNI 192.168.206.10@tcp [ 1758.023455] Key type .llcrypt unregistered [ 1758.030956] Key type ._llcrypt unregistered [ 1764.190802] Key type ._llcrypt registered [ 1764.197356] Key type .llcrypt registered [ 1764.240386] LNet: 92894:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1764.240386] mask usage: [+|-] ... [ 1764.394459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1764.413340] alg: No test for adler32 (adler32-zlib) [ 1765.359708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1765.395612] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1772.428215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 1777.636228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.110@tcp [ 1777.974856] LNet: 93082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.987771] LNetError: 93082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.009350] LNet: Removed LNI 192.168.206.10@tcp [ 1778.469298] Key type .llcrypt unregistered [ 1778.474755] Key type ._llcrypt unregistered [ 1791.139213] Key type ._llcrypt registered [ 1791.142988] Key type .llcrypt registered [ 1792.521483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1794.298220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1796.108980] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:16:59 (1787397419) [ 1797.747600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1799.898759] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:17:03 (1787397423) [ 1802.320989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg610-server [ 1804.104506] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:17:07 (1787397427) [ 1805.751749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg610-server [ 1807.773221] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:17:11 (1787397431) [ 1808.667985] LNet: 95130:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1808.667985] mask usage: [+|-] ... [ 1809.320828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.347322] alg: No test for adler32 (adler32-zlib) [ 1810.385419] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 1810.668165] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1812.360117] Key type lgssc registered [ 1813.521735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.925477] Key type lgssc unregistered [ 1834.253108] LNet: 96259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.277380] LNetError: 96259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.304181] LNet: Removed LNI 192.168.206.10@tcp [ 1835.166617] Key type .llcrypt unregistered [ 1835.174407] Key type ._llcrypt unregistered [ 1852.373377] Key type ._llcrypt registered [ 1852.377589] Key type .llcrypt registered [ 1853.795700] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:17:57 (1787397477) [ 1854.267939] LNet: 97069:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1854.267939] mask usage: [+|-] ... [ 1854.480349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1854.505978] alg: No test for adler32 (adler32-zlib) [ 1855.482260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.533583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1855.569564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1855.608407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1855.656519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1862.173465] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:18:06 (1787397486) [ 1862.387789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.408909] LNet: 97686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.427877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.491880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1862.537327] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1863.358159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.386875] LNet: 97830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.402141] LNetError: 97830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.416777] LNet: Removed LNI 192.168.206.10@tcp [ 1863.459611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.519918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1863.539348] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1863.993968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.017736] LNetError: 97858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.033922] LNet: Removed LNI 192.168.206.10@tcp [ 1864.044806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.086370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1864.112671] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1869.675672] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:18:13 (1787397493) [ 1869.865228] LNet: 98448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.880233] LNet: 98448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1869.896372] LNetError: 98448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.915464] LNet: Removed LNI 192.168.206.10@tcp [ 1870.554407] Key type .llcrypt unregistered [ 1870.557748] Key type ._llcrypt unregistered [ 1876.918186] Key type ._llcrypt registered [ 1876.922314] Key type .llcrypt registered [ 1877.020609] LNet: 98680:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1877.020609] mask usage: [+|-] ... [ 1877.263191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1877.291438] alg: No test for adler32 (adler32-zlib) [ 1878.200516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.283334] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1878.285748] LNet: *** cfs_fail_loc=e003, val=100*** [ 1879.038084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1879.062310] LNet: *** cfs_fail_loc=e003, val=100*** [ 1879.064527] LNet: Skipped 2 previous similar messages [ 1879.074082] Ping reply received with mapping: [ 1879.076584] found metadata reply NI[192.168.206.10@tcp] [ 1879.084923] 0a ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1879.089317] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1879.092927] 5a 5a 5a 5a 5a 5a 5a 5a [ 1879.240968] LNet: 98846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.251373] LNetError: 98846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.267513] LNet: Removed LNI 192.168.206.10@tcp [ 1879.868744] Key type .llcrypt unregistered [ 1879.870834] Key type ._llcrypt unregistered [ 1890.192530] Key type ._llcrypt registered [ 1890.195513] Key type .llcrypt registered [ 1891.555502] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:18:35 (1787397515) [ 1892.052518] Key type .llcrypt unregistered [ 1892.055663] Key type ._llcrypt unregistered [ 1898.942829] Key type ._llcrypt registered [ 1898.950330] Key type .llcrypt registered [ 1899.025973] LNet: 99841:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1899.025973] mask usage: [+|-] ... [ 1899.304960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1899.325884] alg: No test for adler32 (adler32-zlib) [ 1900.441409] LNetError: 99891:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1900.859355] Key type .llcrypt unregistered [ 1900.860929] Key type ._llcrypt unregistered [ 1906.571581] Key type ._llcrypt registered [ 1906.573616] Key type .llcrypt registered [ 1907.818481] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:18:51 (1787397531) [ 1908.551512] Key type .llcrypt unregistered [ 1908.554238] Key type ._llcrypt unregistered [ 1915.041940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1915.050958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1917.650356] Key type ._llcrypt registered [ 1917.652417] Key type .llcrypt registered [ 1917.688919] LNet: 100759:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1917.688919] mask usage: [+|-] ... [ 1917.877496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1917.893589] alg: No test for adler32 (adler32-zlib) [ 1918.786554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.841033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1927.879497] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:19:11 (1787397551) [ 1928.208251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.248455] LNet: 101672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.255488] LNetError: 101672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1928.274444] LNet: Removed LNI 10.1.2.3@tcp [ 1928.299960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.228530] Key type .llcrypt unregistered [ 1929.233387] Key type ._llcrypt unregistered [ 1935.180724] Key type ._llcrypt registered [ 1935.186323] Key type .llcrypt registered [ 1935.217244] LNet: 101903:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1935.217244] mask usage: [+|-] ... [ 1935.417129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1935.435740] alg: No test for adler32 (adler32-zlib) [ 1936.403886] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1936.455742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1936.651865] LNet: 101960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.665688] LNetError: 101960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.678261] LNet: Removed LNI 192.168.206.10@tcp [ 1937.292272] Key type .llcrypt unregistered [ 1937.294155] Key type ._llcrypt unregistered [ 1947.644615] Key type ._llcrypt registered [ 1947.648369] Key type .llcrypt registered [ 1948.577338] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:19:32 (1787397572) [ 1949.250396] Key type .llcrypt unregistered [ 1949.251846] Key type ._llcrypt unregistered [ 1956.610751] Key type ._llcrypt registered [ 1956.612349] Key type .llcrypt registered [ 1956.633859] LNet: 102943:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1956.633859] mask usage: [+|-] ... [ 1956.805136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1956.817809] alg: No test for adler32 (adler32-zlib) [ 1964.910245] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:19:48 (1787397588) [ 1965.241805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.302757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.363723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.383271] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1966.560777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.617929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1966.681941] LNet: Removed LNI 192.168.206.10@tcp [ 1966.988664] LNet: 103702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.524308] Key type .llcrypt unregistered [ 1967.526412] Key type ._llcrypt unregistered [ 1979.378000] Key type ._llcrypt registered [ 1979.381620] Key type .llcrypt registered [ 1980.656372] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:20:04 (1787397604) [ 1980.951221] LNet: 104467:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1980.951221] mask usage: [+|-] ... [ 1981.304630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1981.340817] alg: No test for adler32 (adler32-zlib) [ 1982.270220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.315693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.337260] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 1983.145209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 1983.187225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.10@tcp [ 1983.212121] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1983.216221] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1983.220797] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1983, 0, 0 [ 1983.228139] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1983.230898] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1983.234078] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1983.238548] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1983, 0, 0 [ 1983.243104] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1983.245872] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1983.249351] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1983.251845] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1983, 0, 2033 [ 1983.256467] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1983.259224] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1983.261639] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1983.264572] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1983, 0, 2033 [ 1983.269343] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1988.990516] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:20:12 (1787397612) [ 1989.433577] LNet: 105241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.446501] LNetError: 105241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.460327] LNet: Removed LNI 192.168.206.10@tcp [ 1990.147492] Key type .llcrypt unregistered [ 1990.150692] Key type ._llcrypt unregistered [ 1996.511490] Key type ._llcrypt registered [ 1996.516686] Key type .llcrypt registered [ 1996.565433] LNet: 105465:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1996.565433] mask usage: [+|-] ... [ 1996.800613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.829846] alg: No test for adler32 (adler32-zlib) [ 1997.802157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1997.829802] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2004.648465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 2013.404346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.443537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2013.471494] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 2014.410065] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2014.550542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2014.563289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2014.574842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2016.965686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2016.990128] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2026.843213] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:20:50 (1787397650) [ 2027.122563] LNet: 106748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.133180] LNetError: 106748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.139567] LNet: Removed LNI 192.168.206.10@tcp [ 2028.191215] LNet: Removed LNI 192.168.206.10@tcp1 [ 2028.194422] LNetError: 106748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2028.200165] LNetError: 106748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2028.229236] LNet: Removed LNI 10.1.2.3@tcp99 [ 2028.953508] Key type .llcrypt unregistered [ 2028.957833] Key type ._llcrypt unregistered [ 2034.562359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2034.567494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2037.065277] Key type ._llcrypt registered [ 2037.068565] Key type .llcrypt registered [ 2037.109273] LNet: 107013:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2037.109273] mask usage: [+|-] ... [ 2037.254906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2037.267917] alg: No test for adler32 (adler32-zlib) [ 2038.212529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.250132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.277839] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2038.895549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2038.922674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2040.940745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2040.983658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2041.017697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2041.033811] LNetError: 107492:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2041.065849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2041.099870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2041.414390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2041.480855] LNet: 107512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.486753] LNetError: 107512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2041.492295] LNet: Removed LNI 10.1.2.3@tcp [ 2042.537972] LNet: Removed LNI 192.168.206.10@tcp [ 2043.329461] Key type .llcrypt unregistered [ 2043.335569] Key type ._llcrypt unregistered [ 2054.904545] Key type ._llcrypt registered [ 2054.910461] Key type .llcrypt registered [ 2056.532420] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:21:20 (1787397680) [ 2056.755186] LNet: 108305:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2056.755186] mask usage: [+|-] ... [ 2057.023454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.049325] alg: No test for adler32 (adler32-zlib) [ 2058.022759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.109273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.127509] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2058.967479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.10@tcp [ 2059.063240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg610-client@tcp [ 2065.429386] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:21:29 (1787397689) [ 2065.804802] LNet: 109046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.815438] LNetError: 109046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.832540] LNet: Removed LNI 192.168.206.10@tcp [ 2066.453740] Key type .llcrypt unregistered [ 2066.459486] Key type ._llcrypt unregistered [ 2072.285239] Key type ._llcrypt registered [ 2072.289417] Key type .llcrypt registered [ 2072.443016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.457258] alg: No test for adler32 (adler32-zlib) [ 2073.348547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2073.390272] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2073.425821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2079.213373] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:21:43 (1787397703) [ 2080.774268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2082.422980] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:21:46 (1787397706) [ 2083.949616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2085.361873] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:21:49 (1787397709) [ 2085.592728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.617210] LNet: 110465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.625153] LNetError: 110465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.642829] LNet: Removed LNI 192.168.206.10@tcp [ 2085.674816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.725626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2085.755147] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2086.454968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2086.515172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2092.163295] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:21:56 (1787397716) [ 2092.444328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.458576] LNet: 111174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.467618] LNetError: 111174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.495366] LNet: Removed LNI 192.168.206.10@tcp [ 2092.517872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.553696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2092.576399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2093.140556] Key type .llcrypt unregistered [ 2093.143412] Key type ._llcrypt unregistered [ 2102.642403] Key type ._llcrypt registered [ 2102.644473] Key type .llcrypt registered [ 2103.643651] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:22:07 (1787397727) [ 2103.903746] LNet: 111981:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2103.903746] mask usage: [+|-] ... [ 2104.025583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.038489] alg: No test for adler32 (adler32-zlib) [ 2104.913720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.986056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.005944] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2105.741593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2105.820605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2113.200866] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:22:16 (1787397736) [ 2113.282940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.338956] LNet: 112722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.348932] LNetError: 112722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.379678] LNet: Removed LNI 192.168.206.10@tcp [ 2113.438935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.546665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2113.579755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2120.082918] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:22:23 (1787397743) [ 2120.190981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.258135] LNet: 113303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.322493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.398894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2120.843628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2120.887791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2120.971454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2121.306254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2121.325385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2121.420162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.450957] LNet: 113343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.461873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.503903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2121.930490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2122.024552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2122.078336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2122.381290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2122.404650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.647486] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:22:32 (1787397752) [ 2128.741078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.762685] LNet: 113937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.803897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.882205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.922378] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2129.671947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.690183] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2130.649729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2130.703803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2130.910088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.931655] LNetError: 114222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.936864] LNet: Removed LNI 192.168.206.10@tcp [ 2130.950682] LNet: Removed LNI 192.168.206.10@tcp2 [ 2130.967403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.038787] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2131.056174] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2131.072550] LNet: 114229:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2131.806433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2132.036618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.052729] LNetError: 114266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.060400] LNetError: 114266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2132.066632] LNet: Removed LNI 192.168.206.10@tcp [ 2132.074926] LNet: Removed LNI 192.168.206.10@tcp2 [ 2132.091463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.138823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.160370] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2132.738657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2132.757417] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2133.830325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2133.876510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2133.960641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2134.338906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.364533] LNet: 114550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.369492] LNet: 114550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2134.373981] LNetError: 114550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.379095] LNetError: 114550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2134.386201] LNet: Removed LNI 192.168.206.10@tcp [ 2135.465603] LNet: Removed LNI 192.168.206.10@tcp2 [ 2135.496469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.564879] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2135.571703] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2135.576703] LNet: 114556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2135.955685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2136.142434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.166861] LNet: Removed LNI 192.168.206.10@tcp [ 2137.190273] LNet: Removed LNI 192.168.206.10@tcp2 [ 2137.234436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.350197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.385752] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2138.201125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2138.227793] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2139.368658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2139.479968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2139.808637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.848902] LNetError: 114873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.855801] LNetError: 114873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2139.861411] LNet: Removed LNI 192.168.206.10@tcp [ 2139.886901] LNet: Removed LNI 192.168.206.10@tcp2 [ 2139.930823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.013655] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2140.017636] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2140.023159] LNet: 114879:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2140.316545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2140.470424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.535284] LNet: Removed LNI 192.168.206.10@tcp [ 2141.559126] LNet: Removed LNI 192.168.206.10@tcp2 [ 2141.597165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.638290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.656091] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2142.351673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2142.367885] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2143.232709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2143.261378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2143.361469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2143.564909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.585250] LNet: 115200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.588388] LNet: 115200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2144.608359] LNet: Removed LNI 192.168.206.10@tcp [ 2145.637165] LNet: Removed LNI 192.168.206.10@tcp2 [ 2145.675662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.756195] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2145.761722] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2145.766986] LNet: 115207:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.214581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2146.357877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.371235] LNet: Removed LNI 192.168.206.10@tcp [ 2147.434114] LNet: Removed LNI 192.168.206.10@tcp2 [ 2147.459167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.524656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.567564] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2148.470394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2148.493932] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2149.627334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2149.668793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2149.780828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2150.227625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.257761] LNetError: 115540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.268471] LNetError: 115540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2150.283079] LNet: Removed LNI 192.168.206.10@tcp [ 2150.291277] LNet: Removed LNI 192.168.206.10@tcp2 [ 2150.351371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.447142] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2150.454754] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2150.464746] LNet: 115546:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2150.999517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2151.366647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.416790] LNet: Removed LNI 192.168.206.10@tcp [ 2152.431903] LNet: Removed LNI 192.168.206.10@tcp2 [ 2152.457817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.518961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.565139] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2153.276804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2153.291226] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2153.994918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2154.055515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2154.093930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2154.147860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2154.495291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.551906] LNet: Removed LNI 192.168.206.10@tcp [ 2156.584417] LNet: Removed LNI 192.168.206.10@tcp2 [ 2156.623947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.689774] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2156.695989] LNet: Added LNI 192.168.206.10@tcp2 [8/256/0/180] [ 2157.159478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2163.193528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2164.844132] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:23:08 (1787397788) [ 2164.948239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.976031] LNet: 116656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.985974] LNet: 116656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2164.996673] LNet: Removed LNI 192.168.206.10@tcp [ 2166.067368] LNet: Removed LNI 192.168.206.10@tcp2 [ 2166.126836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.391396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.415393] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2167.081836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2167.108336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2167.963355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.985524] LNetError: 116938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2167.991694] LNetError: 116938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2167.997071] LNet: Removed LNI 10.1.2.3@tcp [ 2168.011860] LNet: Removed LNI 192.168.206.10@tcp [ 2168.025606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.144585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2168.180966] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2168.196334] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2168.710804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2168.762454] LNet: Removed LNI 192.168.206.10@tcp [ 2168.782649] LNet: Removed LNI 10.1.2.3@tcp [ 2174.895527] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:23:18 (1787397798) [ 2175.160613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.215289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.354615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2175.458103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2175.461528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2175.465383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2177.934821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.973043] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2178.601472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2178.622748] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2180.430398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.10@tcp [ 2180.465547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2180.526127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.10@tcp -m PUT -e local_timeout -r 1 [ 2180.588684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2180.633471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.637191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2180.883354] LNetError: 117604:0:(lib-move.c:4714:lnet_parse()) 192.168.206.10@tcp, src 192.168.206.10@tcp: Dropping PUT (error -108 looking up sender) [ 2181.923626] LNet: Removed LNI 10.1.2.3@tcp [ 2182.955652] LNet: Removed LNI 192.168.206.10@tcp [ 2183.321868] Key type .llcrypt unregistered [ 2183.325239] Key type ._llcrypt unregistered [ 2196.307822] Key type ._llcrypt registered [ 2196.312436] Key type .llcrypt registered [ 2197.760486] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:23:41 (1787397821) [ 2198.021477] LNet: 118832:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2198.021477] mask usage: [+|-] ... [ 2198.204247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.238355] alg: No test for adler32 (adler32-zlib) [ 2199.164226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.463182] LNet: 118915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.063470] Key type .llcrypt unregistered [ 2200.065957] Key type ._llcrypt unregistered [ 2206.057346] Key type ._llcrypt registered [ 2206.060453] Key type .llcrypt registered [ 2206.108815] LNet: 119144:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2206.108815] mask usage: [+|-] ... [ 2206.312384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2206.330790] alg: No test for adler32 (adler32-zlib) [ 2207.379225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2207.436663] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2214.356876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 2214.564958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2215.083673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.110@tcp [ 2215.308352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2215.336332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2215.364737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2215.405296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2215.438341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2215.466872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2215.496538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2215.540937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2215.557640] LNet: 119365:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2215.580644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2215.612817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2215.646829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2215.647265] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.110@tcp [ 2215.684230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2215.729321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2215.773663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2216.461335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2216.687294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2216.708705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2216.731590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2216.755717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2216.788361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2216.824818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2216.858791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2216.905572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2216.933192] LNet: 119482:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2216.959817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2216.994824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2217.026647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2217.056625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2217.094387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2217.151118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2217.845206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.110@tcp [ 2218.083717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2218.118475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2218.165468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2218.187298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2218.214388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2218.245363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2218.267904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2218.288568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2218.300324] LNet: 119599:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.321194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2218.343613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2218.379264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2218.418748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2218.452952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2218.483530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.092698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2219.386221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.438967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2219.480564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.533628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.594814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.634341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2219.662422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2219.686542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2219.740042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2219.789573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2219.815651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2219.846515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2219.883519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2219.939719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2220.340576] LNet: 119762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.346236] LNetError: 119762:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.372365] LNet: Removed LNI 192.168.206.10@tcp [ 2220.712700] Key type .llcrypt unregistered [ 2220.717308] Key type ._llcrypt unregistered [ 2237.162184] Key type ._llcrypt registered [ 2237.164454] Key type .llcrypt registered [ 2238.472486] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:24:22 (1787397862) [ 2238.813243] LNet: 120569:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2238.813243] mask usage: [+|-] ... [ 2239.045909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.056538] alg: No test for adler32 (adler32-zlib) [ 2239.995190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.389476] LNet: 120650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.110398] Key type .llcrypt unregistered [ 2241.116200] Key type ._llcrypt unregistered [ 2246.614824] Key type ._llcrypt registered [ 2246.617864] Key type .llcrypt registered [ 2246.685861] LNet: 120879:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2246.685861] mask usage: [+|-] ... [ 2246.909706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2246.924544] alg: No test for adler32 (adler32-zlib) [ 2247.847134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2247.882683] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2256.500198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.110@tcp [ 2265.368882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.461062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2265.493848] LNet: Added LNI 192.168.206.10@tcp1 [8/256/0/180] [ 2266.647195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2267.548319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.110@tcp [ 2267.816539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2267.846415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2267.881640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2267.927858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2267.977869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2268.025975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2268.068989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2268.108759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2268.140038] LNet: 121302:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2268.160390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2268.212300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2268.257385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2268.295032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2268.336796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2268.376398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2269.085984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2269.341535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.396151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2269.411154] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.110@tcp, 192.168.206.110@tcp1 [ 2269.441721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2269.476896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2269.504778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2269.536986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2269.564826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2269.591379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2269.618966] LNet: 121419:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.643772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2269.700849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2269.766406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.816865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.865945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.910445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2270.495281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.110@tcp [ 2270.821906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.870910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2270.916700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.993177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2271.062399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2271.147516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2271.185899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2271.257943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2271.293984] LNet: 121537:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.327307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2271.379425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.429302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.475976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.510714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.534782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2272.089328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2272.324714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.375484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2272.454354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.511337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2272.542845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2272.590041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2272.642892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2272.688781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2272.731169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2272.789858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2272.831418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2272.862921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2272.890665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2272.922856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2277.841812] LNet: 121735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.856898] LNetError: 121735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.866853] LNet: Removed LNI 192.168.206.10@tcp [ 2277.887099] LNet: Removed LNI 192.168.206.10@tcp1 [ 2278.520902] Key type .llcrypt unregistered [ 2278.523739] Key type ._llcrypt unregistered [ 2295.135311] Key type ._llcrypt registered [ 2295.138676] Key type .llcrypt registered [ 2296.600303] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:25:20 (1787397920) [ 2304.201552] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:25:28 (1787397928) [ 2312.729045] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:25:36 (1787397936) [ 2320.343242] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:25:44 (1787397944) [ 2328.404749] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:25:52 (1787397952) [ 2335.899884] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:25:59 (1787397959) [ 2343.493802] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:26:07 (1787397967) [ 2350.554489] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:26:14 (1787397974) [ 2358.434716] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:26:21 (1787397981) [ 2365.730706] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:26:29 (1787397989) [ 2373.053634] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:26:36 (1787397996) [ 2381.038445] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:26:44 (1787398004) [ 2387.792551] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:26:51 (1787398011) [ 2394.027563] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:26:58 (1787398018) [ 2400.620704] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:27:04 (1787398024) [ 2407.164497] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:27:11 (1787398031) [ 2413.930603] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:27:17 (1787398037) [ 2421.301546] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:27:24 (1787398044) [ 2421.553256] LNet: 132272:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2421.553256] mask usage: [+|-] ... [ 2421.733036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.751137] alg: No test for adler32 (adler32-zlib) [ 2422.678659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.855223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2422.889373] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2424.098778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.125974] LNet: 132443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.143382] LNetError: 132443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.169300] LNet: Removed LNI 192.168.206.10@tcp [ 2424.207903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.295911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2424.320148] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2430.406619] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:27:34 (1787398054) [ 2436.832983] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:27:41 (1787398061) [ 2443.788179] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:27:47 (1787398067) [ 2450.954579] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:27:55 (1787398075) [ 2457.051276] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:28:01 (1787398081) [ 2463.868642] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:28:07 (1787398087) [ 2470.952478] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:28:14 (1787398094) [ 2478.195940] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:28:22 (1787398102) [ 2487.844871] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:28:31 (1787398111) [ 2495.305715] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:28:39 (1787398119) [ 2502.982729] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:28:46 (1787398126) [ 2509.195165] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:28:53 (1787398133) [ 2515.936300] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2279 sec ========== 07:28:59 (1787398139) [ 2516.288756] LNet: 139907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.293173] LNetError: 139907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.307634] LNet: Removed LNI 192.168.206.10@tcp [ 2517.000337] Key type .llcrypt unregistered [ 2517.003859] Key type ._llcrypt unregistered [ 2529.562202] Key type ._llcrypt registered [ 2529.568857] Key type .llcrypt registered [ 2529.807196] LNet: 140380:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2529.807196] mask usage: [+|-] ... [ 2530.034049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2530.051863] alg: No test for adler32 (adler32-zlib) [ 2531.216440] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 2531.432747] LNet: Added LNI 192.168.206.10@tcp [8/256/0/180] [ 2533.095627] Key type lgssc registered [ 2534.219479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2589.720886] Lustre: Mounted lustre-client [ 2589.896233] LNet: 141852:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2589.896233] mask usage: [+|-] ... [ 2594.714786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2602.618867] Lustre: Unmounted lustre-client [ 2631.188602] Key type lgssc unregistered [ 2631.493299] LNet: 142833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.498896] LNetError: 142833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.513974] LNet: Removed LNI 192.168.206.10@tcp [ 2632.151656] Key type .llcrypt unregistered [ 2632.153160] Key type ._llcrypt unregistered