[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449193877 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002396] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.006753] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007045] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009165] LSM: Security Framework initializing [ 0.010073] Yama: becoming mindful. [ 0.011066] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.022539] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027384] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030150] *** VALIDATE tmpfs *** [ 0.031524] *** VALIDATE proc *** [ 0.033271] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.036267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039037] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043518] debug: unmapping init [mem 0xffffffffb9459000-0xffffffffb9460fff] [ 0.045227] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046750] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053019] ... event mask: 000000070000000f [ 0.055288] rcu: Hierarchical SRCU implementation. [ 0.057756] smp: Bringing up secondary CPUs ... [ 0.058651] x86: Booting SMP configuration: [ 0.059039] .... node #0, CPUs: #1 #2 #3 [ 0.063656] smp: Brought up 1 node, 4 CPUs [ 0.065026] smpboot: Max logical packages: 1 [ 0.066026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228122] node 0 deferred pages initialised in 160ms [ 0.232096] devtmpfs: initialized [ 0.233326] x86/mm: Memory block size: 128MB [ 0.236949] gcov: version magic: 0x41383552 [ 0.239257] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244327] pinctrl core: initialized pinctrl subsystem [ 0.246204] [ 0.246787] ************************************************************* [ 0.250018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252012] ** ** [ 0.254012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257014] ** ** [ 0.259013] ** This means that this kernel is built to expose internal ** [ 0.261013] ** IOMMU data structures, which may compromise security on ** [ 0.262011] ** your system. ** [ 0.264012] ** ** [ 0.266013] ** If you see this message and you are not debugging the ** [ 0.268012] ** kernel, report this immediately to your vendor! ** [ 0.269011] ** ** [ 0.271012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273012] ************************************************************* [ 0.275676] NET: Registered protocol family 16 [ 0.276426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.281058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.284044] cpuidle: using governor menu [ 0.286000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.287419] PCI: Using configuration type 1 for base access [ 0.289149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.299173] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.302142] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.306230] cryptd: max_cpu_qlen set to 1000 [ 0.309040] ACPI: Added _OSI(Module Device) [ 0.311023] ACPI: Added _OSI(Processor Device) [ 0.313016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.315020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.321681] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.328532] ACPI: Interpreter enabled [ 0.329059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.330014] ACPI: Using IOAPIC for interrupt routing [ 0.332111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.334443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344249] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354175] acpiphp: Slot [2] registered [ 0.356124] acpiphp: Slot [5] registered [ 0.357096] acpiphp: Slot [6] registered [ 0.358164] acpiphp: Slot [3] registered [ 0.359093] acpiphp: Slot [4] registered [ 0.360097] acpiphp: Slot [7] registered [ 0.362104] acpiphp: Slot [8] registered [ 0.364103] acpiphp: Slot [9] registered [ 0.365104] acpiphp: Slot [10] registered [ 0.367135] acpiphp: Slot [11] registered [ 0.368113] acpiphp: Slot [12] registered [ 0.370178] acpiphp: Slot [13] registered [ 0.372117] acpiphp: Slot [14] registered [ 0.373110] acpiphp: Slot [15] registered [ 0.375127] acpiphp: Slot [16] registered [ 0.376109] acpiphp: Slot [17] registered [ 0.378129] acpiphp: Slot [18] registered [ 0.380129] acpiphp: Slot [19] registered [ 0.382127] acpiphp: Slot [20] registered [ 0.384161] acpiphp: Slot [21] registered [ 0.386125] acpiphp: Slot [22] registered [ 0.387118] acpiphp: Slot [23] registered [ 0.389077] acpiphp: Slot [24] registered [ 0.390116] acpiphp: Slot [25] registered [ 0.392110] acpiphp: Slot [26] registered [ 0.394124] acpiphp: Slot [27] registered [ 0.395268] acpiphp: Slot [28] registered [ 0.397135] acpiphp: Slot [29] registered [ 0.399125] acpiphp: Slot [30] registered [ 0.401114] acpiphp: Slot [31] registered [ 0.402093] PCI host bridge to bus 0000:00 [ 0.404024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.406024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.409031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.412036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.414022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417091] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.422145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425139] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.436059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452760] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.461975] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.471015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.475015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.480646] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.489015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.508015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.517731] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.524013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.529017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.542857] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.552180] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.555500] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.558470] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.561424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.564312] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.570044] iommu: Default domain type: Passthrough [ 0.572632] SCSI subsystem initialized [ 0.575232] ACPI: bus type USB registered [ 0.577183] usbcore: registered new interface driver usbfs [ 0.579137] usbcore: registered new interface driver hub [ 0.581154] usbcore: registered new device driver usb [ 0.584215] pps_core: LinuxPPS API ver. 1 registered [ 0.586018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.589093] PTP clock support registered [ 0.591172] EDAC MC: Ver: 3.0.0 [ 0.594194] PCI: Using ACPI for IRQ routing [ 0.596798] NetLabel: Initializing [ 0.597012] NetLabel: domain hash size = 128 [ 0.599014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.601087] NetLabel: unlabeled traffic allowed by default [ 0.604043] vgaarb: loaded [ 0.606357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.608017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.613420] clocksource: Switched to clocksource kvm-clock [ 0.725622] VFS: Disk quotas dquot_6.6.0 [ 0.727523] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.729888] *** VALIDATE ramfs *** [ 0.730868] *** VALIDATE hugetlbfs *** [ 0.732308] pnp: PnP ACPI init [ 0.734840] pnp: PnP ACPI: found 6 devices [ 0.790069] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.793262] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.795824] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.798716] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.801765] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.804537] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.808050] NET: Registered protocol family 2 [ 0.811268] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.817448] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.822077] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.828786] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.833355] TCP: Hash tables configured (established 65536 bind 65536) [ 0.836951] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.841488] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844736] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.848297] NET: Registered protocol family 1 [ 0.851785] RPC: Registered named UNIX socket transport module. [ 0.854362] RPC: Registered udp transport module. [ 0.856488] RPC: Registered tcp transport module. [ 0.858636] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861498] NET: Registered protocol family 44 [ 0.863579] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866299] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868948] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871726] PCI: CLS 0 bytes, default 64 [ 0.874519] Unpacking initramfs... [ 2.280379] debug: unmapping init [mem 0xffff9a053cc64000-0xffff9a053ffcffff] [ 2.283738] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285983] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.290883] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.817692] Initialise system trusted keyrings [ 2.819733] Key type blacklist registered [ 2.822193] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.833819] zbud: loaded [ 2.839078] *** VALIDATE nfs *** [ 2.840690] *** VALIDATE nfs4 *** [ 2.842864] pstore: using deflate compression [ 2.846396] Platform Keyring initialized [ 2.955774] NET: Registered protocol family 38 [ 2.958205] Key type asymmetric registered [ 2.960069] Asymmetric key parser 'x509' registered [ 2.962235] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.966034] io scheduler mq-deadline registered [ 2.968407] io scheduler kyber registered [ 2.970451] io scheduler bfq registered [ 2.972624] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.976991] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.980509] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.983868] ACPI: Power Button [PWRF] [ 2.989725] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.996797] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.010046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.038675] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.068901] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.074269] Non-volatile memory driver v1.3 [ 3.076807] Linux agpgart interface v0.103 [ 3.107308] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.111154] vda: detected capacity change from 0 to 76496896 [ 3.129271] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.132344] vdb: detected capacity change from 0 to 1073741824 [ 3.139153] libphy: Fixed MDIO Bus: probed [ 3.151919] usbcore: registered new interface driver usbserial_generic [ 3.154917] usbserial: USB Serial support registered for generic [ 3.157672] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.162811] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.164656] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.167826] mousedev: PS/2 mouse device common for all mice [ 3.171122] rtc_cmos 00:05: RTC can wake from S4 [ 3.174246] rtc_cmos 00:05: registered as rtc0 [ 3.175645] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.176979] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.178621] intel_pstate: CPU model not supported [ 3.185280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.185868] hid: raw HID events driver (C) Jiri Kosina [ 3.190694] usbcore: registered new interface driver usbhid [ 3.190778] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.192704] usbhid: USB HID core driver [ 3.192864] drop_monitor: Initializing network drop monitor service [ 3.200496] Initializing XFRM netlink socket [ 3.203216] NET: Registered protocol family 10 [ 3.208371] Segment Routing with IPv6 [ 3.209944] NET: Registered protocol family 17 [ 3.212146] mpls_gso: MPLS GSO support [ 3.218267] RAS: Correctable Errors collector initialized. [ 3.220506] AVX version of gcm_enc/dec engaged. [ 3.222223] AES CTR mode by8 optimization enabled [ 3.303691] sched_clock: Marking stable (3303664649, 0)->(4285283210, -981618561) [ 3.307973] registered taskstats version 1 [ 3.309481] Loading compiled-in X.509 certificates [ 3.311420] zswap: loaded using pool lzo/zbud [ 3.336817] Key type big_key registered [ 3.349600] Key type encrypted registered [ 3.351273] ima: No TPM chip found, activating TPM-bypass! [ 3.353563] ima: Allocated hash algorithm: sha1 [ 3.355056] ima: No architecture policies found [ 3.356566] evm: Initialising EVM extended attributes: [ 3.358463] evm: security.selinux [ 3.359980] evm: security.ima [ 3.361087] evm: security.capability [ 3.362690] evm: HMAC attrs: 0x1 [ 3.365468] rtc_cmos 00:05: setting system clock to 2026-08-22 09:55:57 UTC (1787392557) [ 3.372607] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 3.375643] debug: unmapping init [mem 0xffffffffb9182000-0xffffffffb9458fff] [ 3.384133] Write protecting the kernel read-only data: 28672k [ 3.387077] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 3.390017] debug: unmapping init [mem 0xffffffffb8114000-0xffffffffb81fffff] [ 3.418089] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.427157] systemd[1]: Detected virtualization kvm. [ 3.428847] systemd[1]: Detected architecture x86-64. [ 3.430599] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.454355] systemd[1]: No hostname configured. [ 3.456194] systemd[1]: Set hostname to . [ 3.458298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460949] systemd[1]: Initializing machine ID from random generator. [ 3.508961] random: ln: uninitialized urandom read (6 bytes read) [ 3.615180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.618115] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.624441] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.637978] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.315785] device-mapper: uevent: version 1.0.3 [ 4.318497] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager.[ 4.653873] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.161835] virtio_net virtio0 ens2: renamed from eth0 [ 5.302325] scsi host0: ata_piix [ 5.333049] scsi host1: ata_piix [ 5.334931] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.338225] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.981414] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.869117] random: crng init done [ 9.870563] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.402227] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.587588] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.849452] SELinux: Disabled at runtime. [ 11.912371] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.921948] systemd[1]: Detected virtualization kvm. [ 11.923745] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.395855] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.399560] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.405062] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.409329] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.412182] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.418804] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.424335] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ 12.573866] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.804486] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.117660] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.158438] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.268656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.281664] EDAC sbridge: Ver: 1.1.2 [ 14.285163] Key type dns_resolver registered [ 14.611901] NFS: Registering the id_resolver key type [ 14.614112] Key type id_resolver registered [ 14.615686] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-client login: [ 73.915154] libcfs: loading out-of-tree module taints kernel. [ 74.281737] Key type ._llcrypt registered [ 74.288223] Key type .llcrypt registered [ 75.069433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 75.098194] alg: No test for adler32 (adler32-zlib) [ 75.563786] hrtimer: interrupt took 3312301 ns [ 76.760764] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 77.800463] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 79.696343] Key type lgssc registered [ 81.546108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 198.440184] Lustre: Mounted lustre-client [ 204.210550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.233835] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing check_logdir /tmp/testlogs/ [ 221.963865] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing yml_node [ 224.224696] Lustre: lustre-OST0000-osc-ffff9a0598570800: disconnect after 23s idle [ 226.481155] Lustre: DEBUG MARKER: Client: 2.17.54.109 [ 229.439591] Lustre: DEBUG MARKER: MDS: 2.17.54.109 [ 232.225795] Lustre: DEBUG MARKER: OSS: 2.17.54.109 [ 234.319287] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 05:59:46 EDT 2026 [ 250.507987] Lustre: Unmounted lustre-client [ 278.898584] Key type lgssc unregistered [ 279.197828] LNet: 8383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.208135] LNetError: 8383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.225394] LNet: Removed LNI 192.168.203.50@tcp [ 279.924150] Key type .llcrypt unregistered [ 279.929898] Key type ._llcrypt unregistered [ 305.476891] Key type ._llcrypt registered [ 305.479331] Key type .llcrypt registered [ 305.523838] LNet: 8961:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 305.523838] mask usage: [+|-] ... [ 305.688785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 305.712358] alg: No test for adler32 (adler32-zlib) [ 306.707791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 306.792419] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 306.820818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 311.782513] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.244780] LNet: 9279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.256275] LNetError: 9279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.278773] LNet: Removed LNI 192.168.203.50@tcp [ 313.003487] Key type .llcrypt unregistered [ 313.005671] Key type ._llcrypt unregistered [ 320.293726] Key type ._llcrypt registered [ 320.295949] Key type .llcrypt registered [ 321.679331] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:01:14 (1787392874) [ 322.009320] LNet: 9652:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 322.009320] mask usage: [+|-] ... [ 322.166674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 322.193861] alg: No test for adler32 (adler32-zlib) [ 323.159578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.228770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.249826] LNet: 9690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.270272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.335359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 323.364785] LNet: 9697:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 330.418863] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:01:23 (1787392883) [ 330.615225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.648954] LNet: 10273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.668617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.738915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 337.120816] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:01:29 (1787392889) [ 337.337435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.361479] LNet: 10864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.385432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.455848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 343.964767] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:01:36 (1787392896) [ 344.044396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.060203] LNet: 11444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.087947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.186235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 350.575992] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:01:43 (1787392903) [ 350.821563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.839401] LNet: 12034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.857313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.925146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 351.372729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 357.428629] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:01:50 (1787392910) [ 357.526754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.589644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.695499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 358.105730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 358.599466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 365.597365] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:01:58 (1787392918) [ 365.737338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.760946] LNet: 13215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.773873] LNet: 13215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 365.792817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.871243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 372.193082] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:02:05 (1787392925) [ 372.446250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.493093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.570349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 372.607995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.046867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 373.078284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 373.630264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 373.678632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.144124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 374.184525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 374.743723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 374.802231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.365856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 375.399601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 375.946325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 376.002094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.580760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 376.614436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.053936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.103879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.622411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 377.663294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.163375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.199845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.610555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 378.643752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.973906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.018530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.005901] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:02:18 (1787392938) [ 386.202796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.269604] LNet: 14481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.285048] LNet: 14481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 386.341528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.493789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 386.556989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 393.674803] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:02:25 (1787392945) [ 393.834666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.950953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.138223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.222711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 402.088899] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:02:34 (1787392954) [ 402.279682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.416914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.508395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 402.573771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 410.568114] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:02:43 (1787392963) [ 410.761802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.805676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.913800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 410.947307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 418.029738] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:02:50 (1787392970) [ 418.209156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.316210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.406229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 418.439430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 425.747952] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:02:58 (1787392978) [ 426.009957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.053483] LNet: 17420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.063257] LNet: 17420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 426.103421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.197445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 426.257188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 433.102407] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:03:06 (1787392986) [ 433.365931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.423479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.497720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.788351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.657606] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:03:13 (1787392993) [ 440.752718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.799621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.023272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.240610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.854342] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:03:20 (1787393000) [ 448.230720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.260759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.381814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.631582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.002254] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:03:27 (1787393007) [ 455.180838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.277311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.935537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.163306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.271280] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:03:35 (1787393015) [ 463.384268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.425870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.533498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.609231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.706838] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:03:42 (1787393022) [ 469.878958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.928555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.990966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.047368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.264349] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:03:49 (1787393029) [ 476.727462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.763900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.864724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 484.396902] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:03:56 (1787393036) [ 484.491594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.539835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.675173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 491.750271] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:04:04 (1787393044) [ 492.044451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.105117] LNet: 22858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.113336] LNet: 22858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 492.170082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.305789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 492.880882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 500.967212] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:04:13 (1787393053) [ 501.152738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.250320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.346731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 501.411677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 507.603978] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:04:20 (1787393060) [ 507.705190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.747167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.849821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 507.900968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 514.495271] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:04:27 (1787393067) [ 514.784376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.837485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.944696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 514.987972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 521.037205] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:04:34 (1787393074) [ 521.427190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.475265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.563299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.608858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.335569] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:04:41 (1787393081) [ 528.632350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.708735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.817825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 528.912690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 528.948589] LNetError: 25806:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 529.000636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 529.053785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 534.781176] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:04:47 (1787393087) [ 534.869496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.934032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.026296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0SrURo2tO [ 535.101608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0SrURo2tO [ 535.157897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q0SrURo2tO [ 542.643308] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:04:54 (1787393094) [ 542.794560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.843784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.960843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.054177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.145461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 550.500597] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:05:03 (1787393103) [ 550.961935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.000704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.052983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.101756] LNet: Message forwarding will be enabled in 2s [ 556.194620] LNet: Message forwarding enabled [ 556.641595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.670358] LNet: Message forwarding will stop in 2s to 6s [ 560.982950] LNet: Message forwarding disabled [ 561.146964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 562.333370] LNet: Message forwarding will be enabled in 2s [ 566.112890] LNet: Message forwarding enabled [ 568.727193] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:05:21 (1787393121) [ 568.883536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.150329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.198228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 576.189751] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:05:29 (1787393129) [ 576.318292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.400057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.494208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 576.527558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 576.575592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 576.614408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 576.659044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 576.696653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 576.725374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 576.768987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 576.808473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 576.857909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 576.894953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 576.948732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 576.987955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 577.032520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 577.107604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 577.171361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 577.204447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 577.239435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.280124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 577.315931] LNetError: 28876:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 584.279930] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:05:36 (1787393136) [ 584.365944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.400556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.477413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.158428] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:05:43 (1787393143) [ 592.852232] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg350-server [ 594.710799] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:05:47 (1787393147) [ 594.993197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.042377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.106908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.145293] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 596.787551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[51-55]@tcp [ 596.830512] LNet: 30593:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 596.889212] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 596.906658] LNetError: 30382:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.52@tcp has gone from up to down [ 597.860093] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.54@tcp is being used as a gateway but routing feature is not turned on [ 597.876849] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 598.889078] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 598.907091] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 600.935483] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.54@tcp is being used as a gateway but routing feature is not turned on [ 600.966887] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 603.399678] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:05:56 (1787393156) [ 605.035492] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 605.052739] LNetError: 30382:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 605.383679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg350-server [ 606.995247] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:05:59 (1787393159) [ 607.386771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.432330] LNetError: 31527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.492050] LNet: Removed LNI 192.168.203.50@tcp [ 607.536777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.622967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.651212] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 608.393377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 608.430204] LNet: 31644:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 608.443381] LNet: 31644:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 608.474330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 615.257907] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:06:08 (1787393168) [ 615.398545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.428854] LNetError: 32231:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.455754] LNet: Removed LNI 192.168.203.50@tcp [ 615.511153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.657620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 615.720744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.786174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.853941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.943768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.032023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.102390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 616.194993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.231939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.313766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.389590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.422180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.469540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 616.530266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.556143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.613625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.677828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.708150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.745489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 616.803594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.837515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.906644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.000903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.040122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.085459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 617.160331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.197425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.251197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.325145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.353624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.396315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 617.453370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.490516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.542883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.065025] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:06:16 (1787393176) [ 624.764511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg350-server [ 626.636739] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:06:19 (1787393179) [ 628.583544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg350-server [ 630.296578] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:06:23 (1787393183) [ 630.426370] LNet: 33629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.445955] LNet: 33629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 25 previous similar messages [ 631.123778] Key type .llcrypt unregistered [ 631.126175] Key type ._llcrypt unregistered [ 637.362481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 637.517699] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 637.523653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 637.532461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.959543] Key type ._llcrypt registered [ 639.968743] Key type .llcrypt registered [ 640.009105] LNet: 33910:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 640.009105] mask usage: [+|-] ... [ 640.203695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.223878] alg: No test for adler32 (adler32-zlib) [ 641.120779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.161321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.179887] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 641.825587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.843496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 643.569763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 643.604183] LNet: Removed LNI 10.1.2.3@tcp [ 643.720212] LNet: 34351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.728750] LNetError: 34351:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.742730] LNet: Removed LNI 192.168.203.50@tcp [ 644.370482] Key type .llcrypt unregistered [ 644.379935] Key type ._llcrypt unregistered [ 655.553439] Key type ._llcrypt registered [ 655.560683] Key type .llcrypt registered [ 656.936565] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:06:49 (1787393209) [ 657.453921] Key type .llcrypt unregistered [ 657.460896] Key type ._llcrypt unregistered [ 663.239276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 663.362458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 663.370432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 663.389218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 665.947565] Key type ._llcrypt registered [ 665.950741] Key type .llcrypt registered [ 665.979890] LNet: 35386:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 665.979890] mask usage: [+|-] ... [ 666.114947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 666.129666] alg: No test for adler32 (adler32-zlib) [ 667.061511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.103492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.156371] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 667.875900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 667.891353] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 670.405616] LNet: 35844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.410360] LNetError: 35844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 670.423774] LNet: Removed LNI 10.1.2.3@tcp [ 670.434358] LNet: Removed LNI 192.168.203.50@tcp [ 670.996690] Key type .llcrypt unregistered [ 671.003561] Key type ._llcrypt unregistered [ 681.527753] Key type ._llcrypt registered [ 681.529517] Key type .llcrypt registered [ 682.792946] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:07:15 (1787393235) [ 683.252279] Key type .llcrypt unregistered [ 683.254307] Key type ._llcrypt unregistered [ 689.015481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 689.158483] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 689.181980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 689.210856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 691.868189] Key type ._llcrypt registered [ 691.872717] Key type .llcrypt registered [ 691.920694] LNet: 36885:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 691.920694] mask usage: [+|-] ... [ 692.070132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.095500] alg: No test for adler32 (adler32-zlib) [ 693.109825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.177019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 693.202700] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 695.100629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 695.129254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.532583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 697.562760] LNet: Removed LNI 10.1.2.3@tcp [ 697.582522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 697.620727] LNet: Removed LNI 10.1.2.31@tcp [ 697.650280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.679738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.567664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 698.584441] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 699.613071] LNet: 37768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.624277] LNetError: 37768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 699.635501] LNet: Removed LNI 10.1.2.31@tcp [ 699.651071] LNet: Removed LNI 10.1.2.3@tcp [ 700.406275] Key type .llcrypt unregistered [ 700.411246] Key type ._llcrypt unregistered [ 715.398162] Key type ._llcrypt registered [ 715.402925] Key type .llcrypt registered [ 716.957140] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:07:49 (1787393269) [ 717.268971] LNet: 38559:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 717.268971] mask usage: [+|-] ... [ 717.579515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.617442] alg: No test for adler32 (adler32-zlib) [ 718.584499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.703314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.758575] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 719.795691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.832421] LNet: 38722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.841736] LNetError: 38722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.864337] LNet: Removed LNI 192.168.203.50@tcp [ 719.893546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.966962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.50@tcp [ 719.992739] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 721.038548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.50@tcp [ 721.102959] LNet: Removed LNI 192.168.203.50@tcp [ 721.211647] LNet: 38857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.910109] Key type .llcrypt unregistered [ 721.917220] Key type ._llcrypt unregistered [ 734.476359] Key type ._llcrypt registered [ 734.485122] Key type .llcrypt registered [ 736.415264] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:08:08 (1787393288) [ 739.042224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg350-server [ 740.965379] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:08:13 (1787393293) [ 741.748326] Key type .llcrypt unregistered [ 741.752750] Key type ._llcrypt unregistered [ 747.231271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.368716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.374280] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.384857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.087970] Key type ._llcrypt registered [ 750.092256] Key type .llcrypt registered [ 750.131681] LNet: 40234:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 750.131681] mask usage: [+|-] ... [ 750.268369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.290282] alg: No test for adler32 (adler32-zlib) [ 751.248233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.307191] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 752.073175] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 754.415474] LNet: 40694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.427826] LNetError: 40694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.437545] LNet: Removed LNI 192.168.203.50@tcp1 [ 754.447097] LNet: Removed LNI 10.1.2.3@tcp2 [ 755.038847] Key type .llcrypt unregistered [ 755.045854] Key type ._llcrypt unregistered [ 766.324248] Key type ._llcrypt registered [ 766.332959] Key type .llcrypt registered [ 767.921573] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:08:40 (1787393320) [ 768.162735] LNet: 41513:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 768.162735] mask usage: [+|-] ... [ 768.389695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.423746] alg: No test for adler32 (adler32-zlib) [ 769.331112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.399919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.423524] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 770.431944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.452595] LNet: 41680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.463046] LNetError: 41680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.493057] LNet: Removed LNI 192.168.203.50@tcp [ 770.531796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.610303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 770.625626] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 776.835461] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:08:49 (1787393329) [ 777.041526] LNet: 42273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.050670] LNetError: 42273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.067960] LNet: Removed LNI 192.168.203.50@tcp [ 777.548303] Key type .llcrypt unregistered [ 777.551182] Key type ._llcrypt unregistered [ 783.201723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.364143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.384839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.394725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.202528] Key type ._llcrypt registered [ 786.208466] Key type .llcrypt registered [ 786.297372] LNet: 42549:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 786.297372] mask usage: [+|-] ... [ 786.461156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.486978] alg: No test for adler32 (adler32-zlib) [ 787.461483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.545776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.595086] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 788.534753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.567243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.887747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.905659] LNet: 43015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.915277] LNetError: 43015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 790.922551] LNet: Removed LNI 10.1.2.3@tcp [ 790.942227] LNet: Removed LNI 192.168.203.50@tcp [ 790.986225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.063658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 791.111559] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 791.134449] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.599705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.630854] LNet: 43042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.641402] LNetError: 43042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 791.655548] LNetError: 43042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 791.662432] LNet: Removed LNI 10.1.2.3@tcp [ 791.676631] LNet: Removed LNI 192.168.203.50@tcp [ 791.701050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.764173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 791.800072] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 791.811332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.257516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 792.283341] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 793.417450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.439179] LNet: 43204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.445644] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 793.451737] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 793.457062] LNet: Removed LNI 10.1.2.3@tcp [ 793.461398] LNet: Removed LNI 192.168.203.50@tcp [ 793.470882] LNet: Removed LNI 192.168.203.50@tcp2 [ 793.511423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.597472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 793.642099] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 793.651294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.659189] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 800.548667] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:09:13 (1787393353) [ 800.801813] LNet: 43801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.812900] LNetError: 43801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.817744] LNetError: 43801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 800.823519] LNet: Removed LNI 10.1.2.3@tcp [ 800.825737] LNet: Removed LNI 192.168.203.50@tcp [ 800.837776] LNet: Removed LNI 192.168.203.50@tcp2 [ 801.457528] Key type .llcrypt unregistered [ 801.459803] Key type ._llcrypt unregistered [ 807.339802] Key type ._llcrypt registered [ 807.343730] Key type .llcrypt registered [ 807.400187] LNet: 44026:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 807.400187] mask usage: [+|-] ... [ 807.574493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.592898] alg: No test for adler32 (adler32-zlib) [ 808.487115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.527850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.544581] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 809.471683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.502524] LNet: 44195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.523619] LNetError: 44195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.554152] LNet: Removed LNI 192.168.203.50@tcp [ 809.593734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.660386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 809.713315] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 810.241886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.277466] LNet: 44225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.287392] LNetError: 44225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.314668] LNet: Removed LNI 192.168.203.50@tcp [ 810.334714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.406876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 810.445407] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 811.055653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.079780] LNet: Removed LNI 192.168.203.50@tcp [ 811.099197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.182928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 811.222780] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 811.709035] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.863453] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.885361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.896370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.250712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.269815] LNet: 44332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.273843] LNet: 44332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 814.280633] LNetError: 44332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.285699] LNetError: 44332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 814.301042] LNet: Removed LNI 192.168.203.50@tcp [ 814.340385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.416494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 814.448925] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 821.037365] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:09:33 (1787393373) [ 821.112399] LNet: 44921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.127493] LNetError: 44921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.160665] LNet: Removed LNI 192.168.203.50@tcp [ 821.713575] Key type .llcrypt unregistered [ 821.716775] Key type ._llcrypt unregistered [ 828.511354] Key type ._llcrypt registered [ 828.514567] Key type .llcrypt registered [ 828.572814] LNet: 45154:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 828.572814] mask usage: [+|-] ... [ 828.765886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.787987] alg: No test for adler32 (adler32-zlib) [ 829.721364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.787074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.815477] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 830.601981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.622736] LNet: 45327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.633951] LNetError: 45327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.656305] LNet: Removed LNI 192.168.203.50@tcp [ 830.699883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.773358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 830.810958] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 837.316612] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:09:50 (1787393390) [ 837.575080] LNet: 45919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.585889] LNetError: 45919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.612928] LNet: Removed LNI 192.168.203.50@tcp [ 838.211570] Key type .llcrypt unregistered [ 838.217602] Key type ._llcrypt unregistered [ 844.359831] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.481856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.488911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.495314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.112109] Key type ._llcrypt registered [ 847.114610] Key type .llcrypt registered [ 847.145428] LNet: 46189:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 847.145428] mask usage: [+|-] ... [ 847.306524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.318475] alg: No test for adler32 (adler32-zlib) [ 848.204239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.244573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.282539] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 849.215642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.238477] LNet: 46363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.243956] LNetError: 46363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.264537] LNet: Removed LNI 192.168.203.50@tcp [ 849.316826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.384708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 849.408426] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 856.239866] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:10:09 (1787393409) [ 856.420013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.457750] LNet: 46962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.472289] LNetError: 46962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.499028] LNet: Removed LNI 192.168.203.50@tcp [ 856.521473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.636574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 862.781206] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:10:15 (1787393415) [ 862.987846] LNet: 47541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.650546] Key type .llcrypt unregistered [ 863.654075] Key type ._llcrypt unregistered [ 870.326979] Key type ._llcrypt registered [ 870.329503] Key type .llcrypt registered [ 870.366685] LNet: 47768:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 870.366685] mask usage: [+|-] ... [ 870.577081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.609245] alg: No test for adler32 (adler32-zlib) [ 871.522807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.561479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.602736] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 872.374321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.398463] LNet: 47940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.409283] LNetError: 47940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.433443] LNet: Removed LNI 192.168.203.50@tcp [ 872.449119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.511344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 872.533439] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 879.117502] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:10:31 (1787393431) [ 879.289347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.321181] LNet: 48529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.338047] LNetError: 48529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.368237] LNet: Removed LNI 192.168.203.50@tcp [ 879.395892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.457631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 885.454555] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:10:38 (1787393438) [ 887.070511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 888.482155] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:10:41 (1787393441) [ 889.955859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 891.465257] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:10:44 (1787393444) [ 893.067828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 894.522097] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:10:47 (1787393447) [ 896.062358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 897.815506] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:10:50 (1787393450) [ 899.277370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 901.112402] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:10:53 (1787393453) [ 901.189526] LNet: 50578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.909485] Key type .llcrypt unregistered [ 901.912181] Key type ._llcrypt unregistered [ 908.066579] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 908.217039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 908.224229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 908.232898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.762710] Key type ._llcrypt registered [ 909.766124] Key type .llcrypt registered [ 909.814841] LNet: 50861:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 909.814841] mask usage: [+|-] ... [ 910.017662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.040509] alg: No test for adler32 (adler32-zlib) [ 911.014518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.107386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.140938] LNet: Added LNI 192.168.203.50@tcp [8/512/0/180] [ 911.148581] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 911.155396] LNet: 50899:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 911.247628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.262417] LNet: 50918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.267988] LNetError: 50918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.275631] LNet: Removed LNI 10.1.2.3@tcp [ 911.278741] LNetError: 50918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.300427] LNet: Removed LNI 192.168.203.50@tcp [ 911.323243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.382343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.404831] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 911.474821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.493555] LNet: 50942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.499979] LNetError: 50942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.518817] LNet: Removed LNI 192.168.203.50@tcp [ 911.538412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.627391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.657568] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 911.663238] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.666249] LNet: 50950:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 911.766419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.793667] LNet: 50968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.804046] LNetError: 50968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.809341] LNet: Removed LNI 10.1.2.3@tcp [ 911.812396] LNetError: 50968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.827884] LNet: Removed LNI 192.168.203.50@tcp [ 911.865768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.929538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.953354] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 911.958626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.962658] LNet: 50976:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 912.154744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.173786] LNet: 51002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.179406] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.183599] LNet: Removed LNI 10.1.2.3@tcp [ 912.186477] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.202141] LNet: Removed LNI 192.168.203.50@tcp [ 912.221561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.279103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 912.302171] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 912.306762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 912.398571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.419645] LNet: 51030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.434979] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.446938] LNet: Removed LNI 10.1.2.3@tcp [ 912.448394] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.480595] LNet: Removed LNI 192.168.203.50@tcp [ 912.522243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.614023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 912.663282] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 912.673793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 912.675672] LNet: 51038:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 912.769406] LNet: Removed LNI 10.1.2.3@tcp [ 912.786772] LNet: Removed LNI 192.168.203.50@tcp [ 913.463372] Key type .llcrypt unregistered [ 913.466124] Key type ._llcrypt unregistered [ 925.571441] Key type ._llcrypt registered [ 925.579241] Key type .llcrypt registered [ 927.167888] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:11:19 (1787393479) [ 927.742192] LNet: 51854:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 927.742192] mask usage: [+|-] ... [ 927.906685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 927.925059] alg: No test for adler32 (adler32-zlib) [ 928.804564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.859901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.896904] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 929.700919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.50 [ 929.716995] LNet: 52015:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 929.723272] LNet: Removed LNI 192.168.203.50@tcp [ 935.662891] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:11:28 (1787393488) [ 935.826913] LNet: 52581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.842887] LNetError: 52581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.872419] LNet: Removed LNI 192.168.203.50@tcp [ 936.525306] Key type .llcrypt unregistered [ 936.527679] Key type ._llcrypt unregistered [ 942.366338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.623814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 944.203106] Key type ._llcrypt registered [ 944.207802] Key type .llcrypt registered [ 944.252944] LNet: 52858:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 944.252944] mask usage: [+|-] ... [ 944.424584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.438514] alg: No test for adler32 (adler32-zlib) [ 945.869422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 952.011610] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:11:44 (1787393504) [ 952.360780] LNet: 53489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.370167] LNetError: 53489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.392424] LNet: Removed LNI 10.1.2.3@tcp [ 952.964874] Key type .llcrypt unregistered [ 952.967741] Key type ._llcrypt unregistered [ 958.722630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.947662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.953591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.957719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.660188] Key type ._llcrypt registered [ 961.662640] Key type .llcrypt registered [ 961.697971] LNet: 53763:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 961.697971] mask usage: [+|-] ... [ 961.814825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.826220] alg: No test for adler32 (adler32-zlib) [ 962.702858] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.346385] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:12:02 (1787393522) [ 972.215387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 974.589289] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:12:07 (1787393527) [ 975.019982] LNet: 54674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.052678] LNetError: 54674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 975.088187] LNet: Removed LNI 10.1.2.3@tcp [ 976.072195] Key type .llcrypt unregistered [ 976.076218] Key type ._llcrypt unregistered [ 985.008937] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 985.196778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 985.199752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 985.211305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.045552] Key type ._llcrypt registered [ 988.052662] Key type .llcrypt registered [ 988.102780] LNet: 54948:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 988.102780] mask usage: [+|-] ... [ 988.251417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.265407] alg: No test for adler32 (adler32-zlib) [ 989.248698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.339757] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 997.714809] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:12:30 (1787393550) [ 998.165659] LNet: 55903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.171956] LNetError: 55903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.183980] LNet: Removed LNI 10.1.2.3@tcp [ 998.648331] Key type .llcrypt unregistered [ 998.649951] Key type ._llcrypt unregistered [ 1005.059688] Key type ._llcrypt registered [ 1005.064025] Key type .llcrypt registered [ 1005.131402] LNet: 56126:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1005.131402] mask usage: [+|-] ... [ 1005.347429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1005.362649] alg: No test for adler32 (adler32-zlib) [ 1006.259203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1006.309263] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1014.265622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1014.946615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1014.992481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1015.037753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1015.062629] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1015.337088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.379827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.417761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.446513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.480866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.522522] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1015.530530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.574638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.613566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.638804] LNet: 56341:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.662812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.730972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.802102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.887352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.948739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.014387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.078434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1016.582661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1016.642400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1016.691650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1016.727350] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1017.031949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.079729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.127828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.179629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.215042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.243864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.274573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.313071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.341847] LNet: 56450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.370951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.416814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.476907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.526225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.581667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.610346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.660652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.181930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1018.228525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1018.272990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1018.301231] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1018.318380] LNet: Skipped 1 previous similar message [ 1018.579621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.630380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1018.660050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.690388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.721455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.748791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.794195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.835064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.858778] LNet: 56559:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.892540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.936730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.980701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.022554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.064634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.094792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.144317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1019.733084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1019.779732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1019.821159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1020.083268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.118652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1020.164421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.218393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.258411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.294109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.357855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.408414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.459476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.490630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.530390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.574693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.638607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.689658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.723576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.241504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1021.273606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1021.307813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1021.349167] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1021.358879] LNet: Skipped 1 previous similar message [ 1021.632933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.684830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1021.729765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.767802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.799891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.840786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.901891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.933553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.950311] LNet: 56777:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.955158] LNet: 56777:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1021.971623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.009234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.046608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.109609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.163638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.194859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.226427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1022.744187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1022.777434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1022.809808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1023.082902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.115234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1023.141494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.167106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.205373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.243649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.287117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.346257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.407873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.454533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.503065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.555921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.601731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.646344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.678537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1023.929855] LNet: 56928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.935485] LNetError: 56928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.959744] LNet: Removed LNI 192.168.203.50@tcp [ 1024.408337] Key type .llcrypt unregistered [ 1024.411860] Key type ._llcrypt unregistered [ 1042.029564] Key type ._llcrypt registered [ 1042.035591] Key type .llcrypt registered [ 1043.714654] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:13:15 (1787393595) [ 1044.656373] Key type .llcrypt unregistered [ 1044.660562] Key type ._llcrypt unregistered [ 1051.140672] Key type ._llcrypt registered [ 1051.144430] Key type .llcrypt registered [ 1051.219771] LNet: 57966:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1051.219771] mask usage: [+|-] ... [ 1051.422348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1051.457366] alg: No test for adler32 (adler32-zlib) [ 1052.391656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1052.433811] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1059.555083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1069.716870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.761239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1069.781550] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1071.244847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1071.298837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1071.348549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1071.392918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1071.427323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1071.477567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1071.515322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1071.556970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1071.596802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1071.631308] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1071.640659] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1071.889859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.940254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1071.991244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.045959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.103150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.136881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.172652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.211358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.233595] LNet: 58395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.252590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.302397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.336415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.375471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.404603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.476279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.507131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.004717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1073.029485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1073.059487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1073.092858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1073.151857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1073.225953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1073.265367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1073.300550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1073.338955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1073.357956] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1073.363465] LNet: Skipped 2 previous similar messages [ 1073.573076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.597245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1073.619837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.643599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.670360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.691196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.722512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.749226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.767545] LNet: 58516:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1073.785276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.828230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.857584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.922813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.966364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.016961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.098090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.796303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1074.828907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1074.871168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1074.913265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1074.956258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1074.983939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1075.024945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1075.090177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1075.134770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1075.157047] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1075.165947] LNet: Skipped 2 previous similar messages [ 1075.359102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.387794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.411382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.448541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.489058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.549757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.595619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.632574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.651293] LNet: 58637:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.699640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.751607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.805547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.850427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.886419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.917804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.955038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.574943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1076.625662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1076.693990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1076.777790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1076.826589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1076.867024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1076.917593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1076.970378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1077.032764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1077.317164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.343824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1077.373445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.396860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.428445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.461619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.493552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.528106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.573723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.598917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.633902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.670839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.711881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.766535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.803530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1078.345893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1078.385533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1078.434172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1078.472443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1078.504381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1078.536482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1078.566266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1078.603032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1078.642496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1078.670992] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1078.676429] LNet: Skipped 5 previous similar messages [ 1078.884835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.933325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1078.985160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.019487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.065058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.119710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.167612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.218792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.232096] LNet: 58880:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.241438] LNet: 58880:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1079.257929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.283625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.310388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.338853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.367480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.398666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.432565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.958057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1080.006715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1080.055076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1080.088669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1080.128823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1080.177532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1080.225653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1080.279420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1080.320612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1080.513937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.539934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1080.563972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.589869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.617891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.637082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.666909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.710339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.740302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.812708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.864821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.908537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.953771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.007168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.059089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.235345] LNet: 59077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.250515] LNetError: 59077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.261888] LNet: Removed LNI 192.168.203.50@tcp [ 1086.275828] LNet: Removed LNI 192.168.203.50@tcp1 [ 1086.714836] Key type .llcrypt unregistered [ 1086.718125] Key type ._llcrypt unregistered [ 1102.167921] Key type ._llcrypt registered [ 1102.170272] Key type .llcrypt registered [ 1103.193413] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:14:16 (1787393656) [ 1104.195437] Key type .llcrypt unregistered [ 1104.202746] Key type ._llcrypt unregistered [ 1109.300798] Key type ._llcrypt registered [ 1109.304455] Key type .llcrypt registered [ 1109.372352] LNet: 60113:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1109.372352] mask usage: [+|-] ... [ 1109.535692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1109.545693] alg: No test for adler32 (adler32-zlib) [ 1110.438835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1110.469571] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1118.182028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1118.916370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1118.946288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1118.981631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1119.001212] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1119.197980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.238351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1119.272984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.301088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.333378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.363438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.406308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.446362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.465754] LNet: 60328:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.481803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.513846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.547246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.572497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.606309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.625419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.643471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.141794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1120.174643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1120.199260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1120.211345] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1120.397692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.419732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1120.448081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.473308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.497378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.563650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.612285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.645499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.658941] LNet: 60437:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.676971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.704736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.735513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.761637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.805153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.834334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.862931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.314236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1121.345927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1121.380543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1121.401748] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1121.600952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.630405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1121.680493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.716862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.771401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.814053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.849206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.884849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.904031] LNet: 60547:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.927160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.961238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.994857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.021727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.047432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.080776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.120222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.380491] LNet: 60588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.388090] LNetError: 60588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.426631] LNet: Removed LNI 192.168.203.50@tcp [ 1122.902062] Key type .llcrypt unregistered [ 1122.903863] Key type ._llcrypt unregistered [ 1140.075321] Key type ._llcrypt registered [ 1140.077227] Key type .llcrypt registered [ 1141.365795] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:14:54 (1787393694) [ 1142.391631] Key type .llcrypt unregistered [ 1142.395530] Key type ._llcrypt unregistered [ 1149.121582] Key type ._llcrypt registered [ 1149.125111] Key type .llcrypt registered [ 1149.162817] LNet: 61626:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1149.162817] mask usage: [+|-] ... [ 1149.346965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.369781] alg: No test for adler32 (adler32-zlib) [ 1150.276822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.308086] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1158.335560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1167.682603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.700213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1167.737198] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1168.951274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1168.983405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1169.012794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1169.056982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1169.100650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1169.144616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1169.184401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1169.214180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1169.253668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1169.276860] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1169.284655] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1169.485092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.523411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1169.546774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.575277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.604690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.643734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.680602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.714460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.736605] LNet: 62055:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.756918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.782723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.806778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.840163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.871386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.894453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.925619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.953505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.465271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1170.506557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1170.549996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1170.591982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1170.624382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1170.665464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1170.707309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1170.738583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1170.781404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1170.799402] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1170.805974] LNet: Skipped 2 previous similar messages [ 1171.027214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.060271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.092697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.117938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.144499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.171648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.195766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.242924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.264625] LNet: 62179:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.286915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.318818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.348375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.380619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.411725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.441149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.468659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.498917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.903357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1171.945114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1171.977538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1172.021782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1172.058507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1172.097734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1172.136616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1172.164769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1172.203979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1172.240417] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1172.247435] LNet: Skipped 1 previous similar message [ 1172.450817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.480687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.505540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.524790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.550590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.572741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.596286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.622670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.638452] LNet: 62302:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.658955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.687307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.714304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.745685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.777727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.806765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.839722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.879906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.161395] LNet: 62380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.167451] LNetError: 62380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.173702] LNet: Removed LNI 192.168.203.50@tcp [ 1177.191863] LNet: Removed LNI 192.168.203.50@tcp1 [ 1177.513349] Key type .llcrypt unregistered [ 1177.522166] Key type ._llcrypt unregistered [ 1190.393767] Key type ._llcrypt registered [ 1190.399596] Key type .llcrypt registered [ 1191.539207] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:15:44 (1787393744) [ 1193.138120] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1194.577308] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:15:47 (1787393747) [ 1195.417395] Key type .llcrypt unregistered [ 1195.419996] Key type ._llcrypt unregistered [ 1200.499350] Key type ._llcrypt registered [ 1200.505762] Key type .llcrypt registered [ 1200.553742] LNet: 63708:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1200.553742] mask usage: [+|-] ... [ 1200.748805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.764836] alg: No test for adler32 (adler32-zlib) [ 1201.626132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1201.654390] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1208.458081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1209.132790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1209.185282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1209.224460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1209.244280] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1209.480623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.528561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1209.593943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.628778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.675617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.712291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.744958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.777307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.797614] LNet: 63923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.816266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.853353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.883329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.889093] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1209.936811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.975613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.017858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.320794] LNet: 63962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.332446] LNetError: 63962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.355319] LNet: Removed LNI 192.168.203.50@tcp [ 1210.818417] Key type .llcrypt unregistered [ 1210.820741] Key type ._llcrypt unregistered [ 1226.959276] Key type ._llcrypt registered [ 1226.964042] Key type .llcrypt registered [ 1227.032967] LNet: 64412:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1227.032967] mask usage: [+|-] ... [ 1227.283430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.312808] alg: No test for adler32 (adler32-zlib) [ 1228.213303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.231154] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1234.482604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1241.313685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.353916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.375892] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1242.730963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1242.770100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1242.827648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1242.881644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1242.935650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1242.989526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1243.029496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1243.088959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1243.129714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1243.150911] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1243.359566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.385181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1243.413287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.467438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.504731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.544784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.597883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.623577] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1243.627476] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1243.640130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.676346] LNet: 64842:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.702596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.743427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.768921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.796790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.842908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.883879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.724867] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1248.052761] LNet: 64915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.060253] LNetError: 64915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.064664] LNet: Removed LNI 192.168.203.50@tcp [ 1248.073834] LNet: Removed LNI 192.168.203.50@tcp1 [ 1248.466350] Key type .llcrypt unregistered [ 1248.470498] Key type ._llcrypt unregistered [ 1264.072211] Key type ._llcrypt registered [ 1264.081117] Key type .llcrypt registered [ 1265.189886] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:16:58 (1787393818) [ 1265.405735] LNet: 65717:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1265.405735] mask usage: [+|-] ... [ 1265.517341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.532542] alg: No test for adler32 (adler32-zlib) [ 1266.486286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.542691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1266.568780] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1267.212443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.228786] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1267.980258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1268.041259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1268.099903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1268.159703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1268.208594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1268.243555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1268.290302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1268.345752] LNetError: 66016:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1268.576358] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1268.580069] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1270.626111] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1271.851862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1274.720748] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1275.305921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1275.339869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1275.416031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1275.445669] LNet: 66124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.457958] LNetError: 66124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.467838] LNet: Removed LNI 192.168.203.50@tcp [ 1275.481612] LNet: Removed LNI 192.168.203.50@tcp1 [ 1275.509281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.573434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.611575] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1276.377950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1276.397910] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1277.148624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1277.225661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.288063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1277.360621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1277.397270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1277.468499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1277.524861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1277.557491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1277.581909] LNetError: 66396:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1277.604077] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1279.937530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1283.751094] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1283.757609] LNet: Skipped 1 previous similar message [ 1289.046721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1289.079970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.108602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.987723] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:17:28 (1787393848) [ 1296.122458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.156240] LNet: 67108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.163984] LNetError: 67108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.173682] LNetError: 67108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1296.190219] LNet: Removed LNI 192.168.203.50@tcp [ 1296.216164] LNet: Removed LNI 192.168.203.50@tcp1 [ 1296.289793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.342792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.360671] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1296.961597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.983048] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1297.807975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1297.896930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1297.963625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1298.040363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1298.082484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1298.159265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1298.212452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1298.341525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1298.403488] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.50@tcp [ 1299.425873] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1299.444139] LNet: Skipped 1 previous similar message [ 1313.914941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.096994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1314.137387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.175661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 500 [ 1314.275116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.048909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.110742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.126141] LNet: 67613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.131873] LNetError: 67613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.136302] LNetError: 67613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1323.168656] LNet: Removed LNI 192.168.203.50@tcp [ 1323.186340] LNet: Removed LNI 192.168.203.50@tcp1 [ 1323.213616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.268322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1323.304678] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1324.056943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1324.102433] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1325.044480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1325.097913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.172966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1325.213443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1325.243265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1325.301760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1325.355748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1325.389970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1326.368674] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1326.378169] LNet: Skipped 3 previous similar messages [ 1326.789641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.957182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1337.006247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1337.031946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.426697] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:18:16 (1787393896) [ 1344.031159] LNet: 68659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.037348] LNetError: 68659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.044678] LNetError: 68659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1344.049476] LNet: Removed LNI 192.168.203.50@tcp [ 1344.060728] LNet: Removed LNI 192.168.203.50@tcp1 [ 1344.750640] Key type .llcrypt unregistered [ 1344.756453] Key type ._llcrypt unregistered [ 1351.582690] Key type ._llcrypt registered [ 1351.585287] Key type .llcrypt registered [ 1351.655942] LNet: 68882:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1351.655942] mask usage: [+|-] ... [ 1351.916793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1351.946923] alg: No test for adler32 (adler32-zlib) [ 1352.842378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1352.866083] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1361.706727] Lustre: DEBUG MARKER: Initial discovery [ 1361.773416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1364.202836] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1364.381510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1364.423112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1364.445570] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1364.462544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1364.527570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -l 3 [ 1364.598330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -e local_error [ 1364.645856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m PUT -l 6 [ 1365.154341] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1366.124249] Lustre: DEBUG MARKER: Force oleg350-client.virtnet to discover 192.168.203.150@tcp (in background) [ 1366.175047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1367.522654] Lustre: DEBUG MARKER: Force oleg350-server to discover 192.168.203.50@tcp [ 1369.260842] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1370.050843] Lustre: DEBUG MARKER: Wait for 69413 [ 1371.837728] Lustre: DEBUG MARKER: Finished wait on 69413 [ 1371.898290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1371.943186] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1371.950612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1372.240972] LNet: 69873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.245503] LNetError: 69873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.261451] LNet: Removed LNI 192.168.203.50@tcp [ 1372.685548] Key type .llcrypt unregistered [ 1372.691354] Key type ._llcrypt unregistered [ 1389.275299] Key type ._llcrypt registered [ 1389.278315] Key type .llcrypt registered [ 1390.762335] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:19:03 (1787393943) [ 1391.226589] Key type .llcrypt unregistered [ 1391.231895] Key type ._llcrypt unregistered [ 1397.387153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1397.615206] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1397.622645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1397.630284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1400.112450] Key type ._llcrypt registered [ 1400.114589] Key type .llcrypt registered [ 1400.148552] LNet: 70916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1400.148552] mask usage: [+|-] ... [ 1400.292416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1400.319225] alg: No test for adler32 (adler32-zlib) [ 1401.213192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.260937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.279668] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1402.081756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1402.124811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1410.220871] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:19:22 (1787393962) [ 1410.515909] LNet: 71950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.530580] LNetError: 71950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1410.545726] LNet: Removed LNI 10.1.2.3@tcp [ 1410.567262] LNet: Removed LNI 192.168.203.50@tcp [ 1411.204271] Key type .llcrypt unregistered [ 1411.208680] Key type ._llcrypt unregistered [ 1417.595604] Key type ._llcrypt registered [ 1417.611425] Key type .llcrypt registered [ 1417.665372] LNet: 72200:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1417.665372] mask usage: [+|-] ... [ 1417.847815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.865383] alg: No test for adler32 (adler32-zlib) [ 1418.843435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.973613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.026821] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1419.657782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1419.677538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1420.737320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.50@tcp [ 1421.856226] LNetError: Unexpected error -22 connecting to 192.168.203.50@tcp at host 192.168.203.50:988 [ 1427.176698] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:19:40 (1787393980) [ 1427.525719] LNet: 73083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.536634] LNetError: 73083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1427.552597] LNet: Removed LNI 10.1.2.3@tcp [ 1427.579232] LNet: Removed LNI 192.168.203.50@tcp [ 1428.217308] Key type .llcrypt unregistered [ 1428.219580] Key type ._llcrypt unregistered [ 1434.495268] Key type ._llcrypt registered [ 1434.500433] Key type .llcrypt registered [ 1434.553387] LNet: 73314:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1434.553387] mask usage: [+|-] ... [ 1434.701753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.717347] alg: No test for adler32 (adler32-zlib) [ 1435.670512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.741676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1435.765415] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1436.441306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1436.462906] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 1437.291889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.50@tcp1 --nid 192.168.203.50@tcp2 [ 1441.420850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp2 [ 1441.455839] LNetError: 73918:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.50@tcp1 to 192.168.203.50@tcp2 - Route table may be misconfigured [ 1441.484184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp2 192.168.203.50@tcp1 [ 1447.272550] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:20:00 (1787394000) [ 1447.678627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.707298] LNet: 74492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.717407] LNetError: 74492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.724731] LNet: Removed LNI 192.168.203.50@tcp1 [ 1448.756410] LNet: Removed LNI 192.168.203.50@tcp2 [ 1448.804393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.867189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.889901] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1449.630312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.649342] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1450.705518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1450.753934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.50@tcp [ 1450.791141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp1 [ 1450.837253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp -e network_timeout [ 1450.928467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -e network_timeout [ 1450.982242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -e network_timeout [ 1451.019754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -e network_timeout [ 1451.087509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1451.136943] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1451.169618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1451.281817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.386083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.936103] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1457.549386] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:20:10 (1787394010) [ 1457.669949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.697695] LNet: 75347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.711245] LNetError: 75347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.723947] LNetError: 75347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1457.735808] LNet: Removed LNI 192.168.203.50@tcp [ 1457.759193] LNet: Removed LNI 192.168.203.50@tcp1 [ 1457.805197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.930955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1458.698820] Key type .llcrypt unregistered [ 1458.700778] Key type ._llcrypt unregistered [ 1470.598832] Key type ._llcrypt registered [ 1470.611444] Key type .llcrypt registered [ 1471.967812] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:20:24 (1787394024) [ 1472.226710] LNet: 76165:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1472.226710] mask usage: [+|-] ... [ 1472.397344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.410357] alg: No test for adler32 (adler32-zlib) [ 1473.379582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.943602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1476.889293] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:20:29 (1787394029) [ 1477.026820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.077515] LNet: 76485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.105971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.157582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.191657] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1477.937391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1477.951890] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1479.421490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1479.543527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp1 [ 1479.622511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp1 [ 1487.302406] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:20:40 (1787394040) [ 1489.732946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg350-server [ 1492.038430] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:20:44 (1787394044) [ 1494.936791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg350-server [ 1497.029562] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:20:49 (1787394049) [ 1499.112638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg350-server [ 1500.435259] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:20:53 (1787394053) [ 1502.545643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg350-server [ 1504.942857] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:20:57 (1787394057) [ 1507.337565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg350-server [ 1509.204778] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:21:01 (1787394061) [ 1510.809478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg350-server [ 1512.757607] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:21:05 (1787394065) [ 1515.005279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg350-server [ 1516.735309] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:21:09 (1787394069) [ 1518.638385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg350-server [ 1520.272043] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:21:13 (1787394073) [ 1521.992303] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg350-server [ 1523.779728] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:21:16 (1787394076) [ 1524.076246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.117901] LNet: 80370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.139072] LNetError: 80370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.147801] LNet: Removed LNI 192.168.203.50@tcp [ 1525.224198] LNet: Removed LNI 192.168.203.50@tcp1 [ 1525.284731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.376338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.416394] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1526.257963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1526.364866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1526.556935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.578589] LNet: 80518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.585604] LNetError: 80518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.596257] LNetError: 80518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1526.615229] LNet: Removed LNI 192.168.203.50@tcp [ 1526.646121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.709695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1526.742444] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1527.531923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1527.733305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.757381] LNetError: 80663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.780813] LNet: Removed LNI 192.168.203.50@tcp [ 1527.794340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.853392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.875954] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1528.838830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1528.890629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1529.087422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.106204] LNet: 80810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.122749] LNet: 80810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1529.154142] LNet: Removed LNI 192.168.203.50@tcp [ 1529.172456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.269522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1529.299318] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1530.197099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1530.425229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.472548] LNetError: 80956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.491376] LNetError: 80956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1530.518549] LNet: Removed LNI 192.168.203.50@tcp [ 1530.537544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.617774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.648203] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1531.518860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1531.639549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1531.921188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.968675] LNet: Removed LNI 192.168.203.50@tcp [ 1533.012135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.098275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1533.140828] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1534.149634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1534.462682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.488867] LNet: 81249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.492521] LNet: 81249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1534.503285] LNetError: 81249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.510514] LNetError: 81249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.528420] LNet: Removed LNI 192.168.203.50@tcp [ 1535.577544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.689139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.747175] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1536.578729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1536.758152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1537.187742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.214931] LNet: Removed LNI 192.168.203.50@tcp [ 1538.250965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.342681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1538.395444] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1539.276380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1539.521593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.583794] LNet: Removed LNI 192.168.203.50@tcp [ 1540.606715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.695539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.736646] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1541.388936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1541.484219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1541.755601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.824624] LNet: Removed LNI 192.168.203.50@tcp [ 1542.862725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.930193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1542.966299] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1543.761192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1544.026747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.052088] LNet: 81834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.057187] LNet: 81834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1544.067181] LNetError: 81834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.075181] LNetError: 81834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1544.101251] LNet: Removed LNI 192.168.203.50@tcp [ 1544.138893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.223232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.260729] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1544.903595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1544.998424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1545.269389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.346540] LNet: Removed LNI 192.168.203.50@tcp [ 1546.414217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.467156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1546.486539] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1547.191835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1548.753959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.800425] LNet: Removed LNI 192.168.203.50@tcp [ 1549.833758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.914410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.947781] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1550.665381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1550.726733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.751160] LNet: Removed LNI 192.168.203.50@tcp [ 1550.777839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.814862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.834751] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1551.604800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1558.399625] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:21:51 (1787394111) [ 1558.771282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.839286] LNet: Removed LNI 192.168.203.50@tcp [ 1558.885385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.988863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1559.023870] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1560.644199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.668740] LNet: 83112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.674587] LNet: 83112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1560.685876] LNetError: 83112:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.693213] LNetError: 83112:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1560.713814] LNet: Removed LNI 192.168.203.50@tcp231 [ 1560.733823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.826648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1560.869828] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1560.881622] LNet: 83120:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1561.277297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1561.304448] LNet: Removed LNI 192.168.203.50@tcp231 [ 1561.328233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1561.353855] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1562.583532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.631965] LNet: Removed LNI 192.168.203.50@tcp231 [ 1562.653804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.747363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1562.778540] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1562.788984] LNet: 83272:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1563.346523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1563.380254] LNet: Removed LNI 192.168.203.50@tcp231 [ 1563.398081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1563.435276] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1564.667331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.699889] LNet: Removed LNI 192.168.203.50@tcp231 [ 1564.722775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.851222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1564.878303] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1564.889133] LNet: 83423:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1565.378446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1565.416868] LNet: Removed LNI 192.168.203.50@tcp231 [ 1565.438175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1565.466981] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1566.794980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.845933] LNet: Removed LNI 192.168.203.50@tcp231 [ 1566.871106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.964485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1566.994097] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1567.001732] LNet: 83575:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.627443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1567.686474] LNet: Removed LNI 192.168.203.50@tcp231 [ 1567.718763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1567.764790] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1577.048605] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:22:09 (1787394129) [ 1577.350292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.388973] LNet: Removed LNI 192.168.203.50@tcp231 [ 1577.424675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.488856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1577.516492] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1578.312420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1578.357180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1578.404861] LNet: Removed LNI 192.168.203.50@tcp232 [ 1578.428077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1578.476464] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1579.280507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1579.322573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1579.349783] LNet: Removed LNI 192.168.203.50@tcp232 [ 1579.374534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1579.402541] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1580.134823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1580.177598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1580.212684] LNet: Removed LNI 192.168.203.50@tcp232 [ 1580.228657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1580.251903] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1581.029763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1581.068473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1581.101833] LNet: Removed LNI 192.168.203.50@tcp232 [ 1581.123882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1587.322224] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:22:20 (1787394140) [ 1588.356631] Key type .llcrypt unregistered [ 1588.364491] Key type ._llcrypt unregistered [ 1593.696877] Key type ._llcrypt registered [ 1593.699339] Key type .llcrypt registered [ 1593.725617] LNet: 85652:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1593.725617] mask usage: [+|-] ... [ 1593.872378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.896356] alg: No test for adler32 (adler32-zlib) [ 1594.826996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1594.855841] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1601.444352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1609.704272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.759664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1609.828598] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1611.671501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1611.720349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1611.777199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1611.832560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1611.890945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1611.948795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1611.992936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1612.071475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1612.127497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1612.170938] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1612.186490] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1612.451215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1612.496313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1612.544494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1612.589787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1612.638526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1612.680552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1612.717821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1612.742712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1612.759813] LNet: 86086:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1612.779220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1612.814452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1612.866822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1612.892733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.919284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1612.951050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1613.518558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1613.560713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1613.610358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1613.643774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1613.693931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1613.747309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1613.785825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1613.825063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1613.855901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1613.878690] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1614.161980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1614.211864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1614.272330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1614.321807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1614.363029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1614.403487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1614.428760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1614.463689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1614.491762] LNet: 86206:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.508492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1614.552650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1614.578274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1614.607134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1614.667370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1614.721790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1615.385100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1615.411367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1615.446822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1615.481855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1615.515386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1615.544754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1615.572293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1615.602552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1615.640476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1615.663614] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1615.900418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.928033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1615.963930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.994169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1616.020823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1616.051925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1616.079459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1616.108714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1616.123646] LNet: 86326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1616.136364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1616.163906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1616.201871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1616.241903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1616.277545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1616.313241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.874752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1616.912538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1616.939871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1616.985689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1617.018142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1617.045848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1617.078726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1617.125167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1617.176763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1617.481326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.515401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1617.539692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.565368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.588491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.614329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.645289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.677466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.706680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.737469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.768472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.800763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.864972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1617.912151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.460722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1618.514259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1618.559712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1618.615415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1618.652586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1618.698363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1618.757814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1618.831442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1618.921905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1618.964118] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1618.973913] LNet: Skipped 1 previous similar message [ 1619.259916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.288120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1619.314307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.369438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.408345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.445899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.479501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.501394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.526433] LNet: 86567:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.539838] LNet: 86567:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1619.569352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.610525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.650976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.689204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.742553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1619.792961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.372779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1620.402774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1620.438514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1620.470722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1620.501784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1620.534193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1620.571261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1620.607357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1620.659875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1620.724295] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1621.021824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.052576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1621.077537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.098976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.157635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.205981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.258563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.301254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.353115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.400948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.458975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.503399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.548920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.588124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.989912] LNet: 86762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.996743] LNetError: 86762:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.004911] LNet: Removed LNI 192.168.203.50@tcp [ 1627.017700] LNet: Removed LNI 192.168.203.50@tcp1 [ 1627.429599] Key type .llcrypt unregistered [ 1627.433171] Key type ._llcrypt unregistered [ 1647.242660] Key type ._llcrypt registered [ 1647.247938] Key type .llcrypt registered [ 1648.951499] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:23:21 (1787394201) [ 1649.297041] LNet: 87569:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1649.297041] mask usage: [+|-] ... [ 1649.507252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.527282] alg: No test for adler32 (adler32-zlib) [ 1650.545778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.601883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.632078] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1651.933676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1651.965261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1651.984605] LNetError: 87741:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1652.071803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.50@tcp [ 1652.189984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.435574] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:23:31 (1787394211) [ 1658.558070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.609905] LNet: 88326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.624131] LNetError: 88326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.656183] LNet: Removed LNI 192.168.203.50@tcp [ 1658.679979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.725796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.746353] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1659.392237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1659.416879] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1660.395336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.50@tcp [ 1660.416855] LNetError: 88596:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1660.770912] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1660.776107] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1661.707318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.50@tcp [ 1669.286912] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:23:42 (1787394222) [ 1671.249587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg350-server [ 1673.066976] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:23:45 (1787394225) [ 1674.957394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg350-server [ 1676.837128] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:23:49 (1787394229) [ 1678.703466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg350-server [ 1680.589320] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:23:53 (1787394233) [ 1680.715536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.739137] LNet: 90215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.745869] LNetError: 90215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.760173] LNet: Removed LNI 192.168.203.50@tcp [ 1681.769737] LNetError: 90215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.803503] LNet: Removed LNI 192.168.203.50@tcp1 [ 1681.834606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.944310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.995590] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1682.709470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1682.765171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1682.810478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1682.876469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1682.926607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1682.971796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1690.009340] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:24:02 (1787394242) [ 1690.337919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.367721] LNet: 90945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.378597] LNetError: 90945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.399614] LNet: Removed LNI 192.168.203.50@tcp [ 1690.431753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.506252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.553285] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1698.329320] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:24:10 (1787394250) [ 1698.528914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.555945] LNet: 91646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.566302] LNetError: 91646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.594076] LNet: Removed LNI 192.168.203.50@tcp [ 1698.637691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.707953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.752615] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1705.870175] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:24:18 (1787394258) [ 1707.385593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1708.910498] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:24:21 (1787394261) [ 1709.443881] LNet: 92683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.452904] LNetError: 92683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.470864] LNet: Removed LNI 192.168.203.50@tcp [ 1710.169442] Key type .llcrypt unregistered [ 1710.174143] Key type ._llcrypt unregistered [ 1716.044547] Key type ._llcrypt registered [ 1716.046875] Key type .llcrypt registered [ 1716.089387] LNet: 92912:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1716.089387] mask usage: [+|-] ... [ 1716.314297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.335293] alg: No test for adler32 (adler32-zlib) [ 1717.258961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1717.288140] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1725.333388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1730.150853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.150@tcp [ 1730.419791] LNet: 93100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.425697] LNetError: 93100:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.444312] LNet: Removed LNI 192.168.203.50@tcp [ 1730.809478] Key type .llcrypt unregistered [ 1730.811411] Key type ._llcrypt unregistered [ 1742.140874] Key type ._llcrypt registered [ 1742.142910] Key type .llcrypt registered [ 1743.435591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1744.874304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1746.438545] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:24:59 (1787394299) [ 1747.590358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1749.098098] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:25:02 (1787394302) [ 1750.656128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg350-server [ 1752.355408] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:25:05 (1787394305) [ 1754.216480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg350-server [ 1755.648744] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:25:08 (1787394308) [ 1757.042697] LNet: 95151:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1757.042697] mask usage: [+|-] ... [ 1757.514870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.542523] alg: No test for adler32 (adler32-zlib) [ 1758.641849] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1758.905730] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1760.600258] Key type lgssc registered [ 1761.741356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1780.076309] Key type lgssc unregistered [ 1780.351536] LNet: 96275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.368993] LNetError: 96275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.392490] LNet: Removed LNI 192.168.203.50@tcp [ 1781.157860] Key type .llcrypt unregistered [ 1781.161184] Key type ._llcrypt unregistered [ 1795.876977] Key type ._llcrypt registered [ 1795.885608] Key type .llcrypt registered [ 1797.348266] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:25:49 (1787394349) [ 1797.599096] LNet: 97070:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1797.599096] mask usage: [+|-] ... [ 1797.770809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.789631] alg: No test for adler32 (adler32-zlib) [ 1798.793164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.887882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1798.977034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1799.038960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1799.112921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1805.885095] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:25:58 (1787394358) [ 1805.994744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.032686] LNet: 97683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.062843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.166449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.231120] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1807.256151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.290175] LNet: 97833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.302315] LNetError: 97833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.320130] LNet: Removed LNI 192.168.203.50@tcp [ 1807.354469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.442679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1807.481513] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1808.037402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.067732] LNetError: 97861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.083629] LNet: Removed LNI 192.168.203.50@tcp [ 1808.125376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.206034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1808.258605] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1815.713310] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:26:08 (1787394368) [ 1815.812799] LNet: 98444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.819416] LNet: 98444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1815.830234] LNetError: 98444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.864721] LNet: Removed LNI 192.168.203.50@tcp [ 1816.542334] Key type .llcrypt unregistered [ 1816.543836] Key type ._llcrypt unregistered [ 1823.318342] Key type ._llcrypt registered [ 1823.320077] Key type .llcrypt registered [ 1823.375493] LNet: 98684:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1823.375493] mask usage: [+|-] ... [ 1823.471744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1823.485985] alg: No test for adler32 (adler32-zlib) [ 1824.447674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.537058] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1824.541217] LNet: *** cfs_fail_loc=e003, val=100*** [ 1825.517270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1825.542399] LNet: *** cfs_fail_loc=e003, val=100*** [ 1825.548306] LNet: Skipped 2 previous similar messages [ 1825.553653] Ping reply received with mapping: [ 1825.555266] found metadata reply NI[192.168.203.50@tcp] [ 1825.562423] 32 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1825.569835] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1825.575469] 5a 5a 5a 5a 5a 5a 5a 5a [ 1825.878383] LNet: 98850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.888674] LNetError: 98850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.911865] LNet: Removed LNI 192.168.203.50@tcp [ 1826.683328] Key type .llcrypt unregistered [ 1826.686547] Key type ._llcrypt unregistered [ 1839.062344] Key type ._llcrypt registered [ 1839.065484] Key type .llcrypt registered [ 1840.454707] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:26:33 (1787394393) [ 1841.083173] Key type .llcrypt unregistered [ 1841.090318] Key type ._llcrypt unregistered [ 1847.480929] Key type ._llcrypt registered [ 1847.483973] Key type .llcrypt registered [ 1847.557764] LNet: 99846:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1847.557764] mask usage: [+|-] ... [ 1847.778529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1847.814419] alg: No test for adler32 (adler32-zlib) [ 1848.957118] LNetError: 99896:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1849.391137] Key type .llcrypt unregistered [ 1849.395228] Key type ._llcrypt unregistered [ 1854.244394] Key type ._llcrypt registered [ 1854.247591] Key type .llcrypt registered [ 1855.774894] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:26:48 (1787394408) [ 1856.426882] Key type .llcrypt unregistered [ 1856.431032] Key type ._llcrypt unregistered [ 1862.454644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1862.459982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1865.003427] Key type ._llcrypt registered [ 1865.007425] Key type .llcrypt registered [ 1865.042073] LNet: 100764:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1865.042073] mask usage: [+|-] ... [ 1865.187324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.214664] alg: No test for adler32 (adler32-zlib) [ 1866.164326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.219050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1875.639562] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:27:08 (1787394428) [ 1875.822940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.886024] LNet: 101671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.897536] LNetError: 101671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1875.917657] LNet: Removed LNI 10.1.2.3@tcp [ 1875.951968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.851111] Key type .llcrypt unregistered [ 1876.854977] Key type ._llcrypt unregistered [ 1883.619672] Key type ._llcrypt registered [ 1883.621721] Key type .llcrypt registered [ 1883.657561] LNet: 101908:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1883.657561] mask usage: [+|-] ... [ 1883.822204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1883.844650] alg: No test for adler32 (adler32-zlib) [ 1884.857259] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1884.927861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1885.117218] LNet: 101965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.125513] LNetError: 101965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.146491] LNet: Removed LNI 192.168.203.50@tcp [ 1885.819780] Key type .llcrypt unregistered [ 1885.825184] Key type ._llcrypt unregistered [ 1897.507382] Key type ._llcrypt registered [ 1897.512476] Key type .llcrypt registered [ 1898.888639] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:27:31 (1787394451) [ 1899.925535] Key type .llcrypt unregistered [ 1899.927552] Key type ._llcrypt unregistered [ 1906.827373] Key type ._llcrypt registered [ 1906.831324] Key type .llcrypt registered [ 1906.864457] LNet: 102949:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1906.864457] mask usage: [+|-] ... [ 1907.019374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.042104] alg: No test for adler32 (adler32-zlib) [ 1914.032505] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:27:46 (1787394466) [ 1914.251202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.292869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.361711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.389644] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1915.275478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.343168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1915.380990] LNet: Removed LNI 192.168.203.50@tcp [ 1915.700356] LNet: 103708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.096465] Key type .llcrypt unregistered [ 1916.104442] Key type ._llcrypt unregistered [ 1927.723609] Key type ._llcrypt registered [ 1927.730673] Key type .llcrypt registered [ 1929.063921] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:28:01 (1787394481) [ 1929.338992] LNet: 104473:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1929.338992] mask usage: [+|-] ... [ 1929.493954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.514884] alg: No test for adler32 (adler32-zlib) [ 1930.472514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.534162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.553926] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1931.335331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1931.391861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.50@tcp [ 1931.417662] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1931.421696] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1931.426515] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1931, 0, 0 [ 1931.430819] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1931.433259] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1931.436312] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1931.438979] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1931, 0, 0 [ 1931.443333] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1931.446288] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1931.452944] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1931.458661] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1931, 0, 1981 [ 1931.464681] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1931.469651] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1931.475484] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1931.481895] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1931, 0, 1981 [ 1931.488942] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1937.930494] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:28:10 (1787394490) [ 1938.700596] LNet: 105249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.707529] LNetError: 105249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.722652] LNet: Removed LNI 192.168.203.50@tcp [ 1939.390402] Key type .llcrypt unregistered [ 1939.394374] Key type ._llcrypt unregistered [ 1946.233512] Key type ._llcrypt registered [ 1946.238041] Key type .llcrypt registered [ 1946.296535] LNet: 105472:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1946.296535] mask usage: [+|-] ... [ 1946.503301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1946.516167] alg: No test for adler32 (adler32-zlib) [ 1947.518913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1947.579519] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1955.611475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1965.330898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.394484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1965.431540] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1966.414418] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1966.576753] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1966.589746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1966.598387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1968.911054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1968.940313] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1979.135766] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:28:52 (1787394532) [ 1979.426574] LNet: 106755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.435726] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.441289] LNet: Removed LNI 192.168.203.50@tcp [ 1980.448129] LNet: Removed LNI 192.168.203.50@tcp1 [ 1980.455083] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1980.467991] LNetError: 106755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1980.490041] LNet: Removed LNI 10.1.2.3@tcp99 [ 1981.214126] Key type .llcrypt unregistered [ 1981.216632] Key type ._llcrypt unregistered [ 1987.252597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1987.259376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.802595] Key type ._llcrypt registered [ 1989.804902] Key type .llcrypt registered [ 1989.827401] LNet: 107021:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1989.827401] mask usage: [+|-] ... [ 1989.954510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1989.964651] alg: No test for adler32 (adler32-zlib) [ 1990.870341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.925560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.953597] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1991.736495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1991.757571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1993.880128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1993.919506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1993.966062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1993.984736] LNetError: 107500:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1994.022720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1994.088555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1994.544705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1994.624179] LNet: 107520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.637285] LNetError: 107520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1994.641636] LNet: Removed LNI 10.1.2.3@tcp [ 1995.686315] LNet: Removed LNI 192.168.203.50@tcp [ 1996.391305] Key type .llcrypt unregistered [ 1996.396218] Key type ._llcrypt unregistered [ 2009.265497] Key type ._llcrypt registered [ 2009.267306] Key type .llcrypt registered [ 2010.616327] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:29:23 (1787394563) [ 2010.883612] LNet: 108319:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2010.883612] mask usage: [+|-] ... [ 2011.040848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.058833] alg: No test for adler32 (adler32-zlib) [ 2011.930322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.999612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2012.019159] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2012.705055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2012.813539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg350-client@tcp [ 2019.013097] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:29:31 (1787394571) [ 2019.292591] LNet: 109054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.303680] LNetError: 109054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.324457] LNet: Removed LNI 192.168.203.50@tcp [ 2020.024299] Key type .llcrypt unregistered [ 2020.030128] Key type ._llcrypt unregistered [ 2026.546299] Key type ._llcrypt registered [ 2026.548801] Key type .llcrypt registered [ 2026.728717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.753339] alg: No test for adler32 (adler32-zlib) [ 2027.617646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2027.638309] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2027.676615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2034.059728] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:29:47 (1787394587) [ 2035.864102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2037.505906] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:29:50 (1787394590) [ 2038.958941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2040.650176] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:29:53 (1787394593) [ 2040.958817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.989578] LNet: 110472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.995576] LNetError: 110472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.014490] LNet: Removed LNI 192.168.203.50@tcp [ 2041.034780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.095294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.122536] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2041.832592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2041.887728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2048.491790] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:30:01 (1787394601) [ 2048.816562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.833644] LNet: 111181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.839637] LNetError: 111181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.861112] LNet: Removed LNI 192.168.203.50@tcp [ 2048.883049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.923219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2048.954278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2049.597328] Key type .llcrypt unregistered [ 2049.604747] Key type ._llcrypt unregistered [ 2060.919464] Key type ._llcrypt registered [ 2060.921985] Key type .llcrypt registered [ 2062.166715] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:30:14 (1787394614) [ 2062.400631] LNet: 111988:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2062.400631] mask usage: [+|-] ... [ 2062.555427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.567398] alg: No test for adler32 (adler32-zlib) [ 2063.456915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.511192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.533661] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2064.337992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2064.390855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2070.293563] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:30:23 (1787394623) [ 2070.655075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.706900] LNet: 112737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.718511] LNetError: 112737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.785335] LNet: Removed LNI 192.168.203.50@tcp [ 2071.837499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.922762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2071.970165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2077.752396] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:30:30 (1787394630) [ 2077.924894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.960190] LNet: 113316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.996545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.078744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2078.444594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2078.475624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2078.526538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2078.961992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2079.002657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2079.101381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.118181] LNet: 113350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.138824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.227812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2079.561431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2079.592569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2079.672917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2080.135966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2080.166225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2086.685862] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:30:39 (1787394639) [ 2086.897270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.937687] LNet: 113951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.974781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.038934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.083111] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2087.841738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2087.865737] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2088.610805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2088.674074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2088.892838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.922305] LNetError: 114230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.952484] LNet: Removed LNI 192.168.203.50@tcp [ 2089.959525] LNetError: 114230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.994937] LNet: Removed LNI 192.168.203.50@tcp2 [ 2090.035756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.119502] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2090.125382] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2090.132961] LNet: 114236:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2090.538468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2090.728324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.808191] LNet: Removed LNI 192.168.203.50@tcp [ 2091.826280] LNet: Removed LNI 192.168.203.50@tcp2 [ 2091.862446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.928127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2091.951892] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2092.632705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2092.667933] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2093.750675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2093.790842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2093.836952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2094.112417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.168940] LNet: 114558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.179679] LNet: 114558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2094.193202] LNetError: 114558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.200239] LNetError: 114558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2094.205465] LNet: Removed LNI 192.168.203.50@tcp [ 2094.229892] LNet: Removed LNI 192.168.203.50@tcp2 [ 2094.296520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.390425] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2094.397336] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2094.401977] LNet: 114564:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2094.668701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2094.900795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.917299] LNet: Removed LNI 192.168.203.50@tcp [ 2095.979782] LNet: Removed LNI 192.168.203.50@tcp2 [ 2096.024324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.140748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.153440] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2097.000792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2097.049349] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2098.143060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2098.207201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2098.501982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.526848] LNetError: 114880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.533353] LNetError: 114880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2098.544034] LNet: Removed LNI 192.168.203.50@tcp [ 2098.563874] LNet: Removed LNI 192.168.203.50@tcp2 [ 2098.606556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.716911] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2098.724816] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2098.734986] LNet: 114886:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.593672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2099.801939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.832174] LNet: Removed LNI 192.168.203.50@tcp [ 2100.854922] LNet: Removed LNI 192.168.203.50@tcp2 [ 2100.889776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.956988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.985989] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2101.786994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2101.816405] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2103.739907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2103.845638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2104.004229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2104.376936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.412734] LNet: 115208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.424414] LNet: 115208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2105.445103] LNet: Removed LNI 192.168.203.50@tcp [ 2106.492313] LNet: Removed LNI 192.168.203.50@tcp2 [ 2106.584738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.705965] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2106.720879] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2106.731929] LNet: 115215:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2107.685662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2107.956565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.008643] LNetError: 115252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.014313] LNetError: 115252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2108.029059] LNet: Removed LNI 192.168.203.50@tcp [ 2109.094862] LNet: Removed LNI 192.168.203.50@tcp2 [ 2109.202498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.317209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.420959] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2110.753872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2110.776264] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2111.600944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2111.623542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2111.664507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2111.998175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.056229] LNet: Removed LNI 192.168.203.50@tcp [ 2113.093911] LNet: Removed LNI 192.168.203.50@tcp2 [ 2113.235656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.328597] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2113.348156] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2113.361151] LNet: 115554:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2114.179372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2114.808765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.872185] LNet: Removed LNI 192.168.203.50@tcp [ 2115.889643] LNet: Removed LNI 192.168.203.50@tcp2 [ 2115.922886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.976308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.007211] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2117.404809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2117.420910] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2118.831514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2118.857654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2118.891888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2119.029841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2119.595263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.673884] LNet: Removed LNI 192.168.203.50@tcp [ 2121.707719] LNet: Removed LNI 192.168.203.50@tcp2 [ 2121.739610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.872212] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2121.881386] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2121.886231] LNet: 115915:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.226802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2129.218576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2130.831574] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:31:23 (1787394683) [ 2131.161637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.193898] LNet: 116674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.202930] LNet: 116674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2131.211932] LNetError: 116674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.222965] LNetError: 116674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2131.234449] LNet: Removed LNI 192.168.203.50@tcp [ 2132.263398] LNet: Removed LNI 192.168.203.50@tcp2 [ 2132.307521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.823792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.881362] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2133.948589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2133.995054] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2135.165703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.200719] LNet: Removed LNI 10.1.2.3@tcp [ 2135.218189] LNet: Removed LNI 192.168.203.50@tcp [ 2135.267337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.367944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2135.417786] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2135.423558] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2135.803756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2135.831550] LNet: Removed LNI 192.168.203.50@tcp [ 2135.847431] LNet: Removed LNI 10.1.2.3@tcp [ 2142.781990] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:31:35 (1787394695) [ 2143.059840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.088412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.259099] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2143.439657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2143.451051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2143.459097] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2145.787142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.831228] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2146.816777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2146.847482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2149.253132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2149.292446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2149.357694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.50@tcp -m PUT -e local_timeout -r 1 [ 2149.468797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2149.533248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2149.536645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2149.785614] LNetError: 117615:0:(lib-move.c:4714:lnet_parse()) 192.168.203.50@tcp, src 192.168.203.50@tcp: Dropping PUT (error -108 looking up sender) [ 2150.824810] LNet: Removed LNI 10.1.2.3@tcp [ 2151.849757] LNet: Removed LNI 192.168.203.50@tcp [ 2152.415676] Key type .llcrypt unregistered [ 2152.421208] Key type ._llcrypt unregistered [ 2165.328572] Key type ._llcrypt registered [ 2165.330897] Key type .llcrypt registered [ 2166.703942] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:31:59 (1787394719) [ 2167.121801] LNet: 118844:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2167.121801] mask usage: [+|-] ... [ 2167.397476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2167.419312] alg: No test for adler32 (adler32-zlib) [ 2168.353728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.861905] LNet: 118926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.576368] Key type .llcrypt unregistered [ 2169.578610] Key type ._llcrypt unregistered [ 2176.129353] Key type ._llcrypt registered [ 2176.131479] Key type .llcrypt registered [ 2176.246237] LNet: 119155:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2176.246237] mask usage: [+|-] ... [ 2176.697819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2176.732125] alg: No test for adler32 (adler32-zlib) [ 2177.817843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2177.878102] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2187.700823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 2188.004944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2188.881314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.150@tcp [ 2189.154828] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 2189.293526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2189.328514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2189.371949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2189.422150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2189.468491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2189.540349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2189.612578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2189.679344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2189.709655] LNet: 119376:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2189.748475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2189.837764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2189.905551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2189.985374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2190.018818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2190.044881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2190.877749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2191.095842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2191.114650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2191.137862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2191.167770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2191.189603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2191.221913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2191.249695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2191.273790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2191.287106] LNet: 119493:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2191.300826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2191.319267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2191.338715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2191.361138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2191.383875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2191.405604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2191.854968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.150@tcp [ 2192.244854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2192.334270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2192.380993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2192.438963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2192.500042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2192.544226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2192.617825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2192.681875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2192.713224] LNet: 119610:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2192.756321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2192.839626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2192.881967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2192.922884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2192.960302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2193.003299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2193.712884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2194.019856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2194.096847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2194.160926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2194.221978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2194.263214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2194.302250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2194.337112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2194.370781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2194.400276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2194.445663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2194.502349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2194.547634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2194.578252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2194.605962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2195.001943] LNet: 119774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.010497] LNetError: 119774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.027653] LNet: Removed LNI 192.168.203.50@tcp [ 2195.520931] Key type .llcrypt unregistered [ 2195.523477] Key type ._llcrypt unregistered [ 2213.616260] Key type ._llcrypt registered [ 2213.622206] Key type .llcrypt registered [ 2215.614611] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:32:47 (1787394767) [ 2216.112374] LNet: 120580:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2216.112374] mask usage: [+|-] ... [ 2216.249345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.258696] alg: No test for adler32 (adler32-zlib) [ 2217.265858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.859911] LNet: 120661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.618623] Key type .llcrypt unregistered [ 2218.625310] Key type ._llcrypt unregistered [ 2228.198430] Key type ._llcrypt registered [ 2228.203477] Key type .llcrypt registered [ 2228.268798] LNet: 120891:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2228.268798] mask usage: [+|-] ... [ 2228.409739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2228.429973] alg: No test for adler32 (adler32-zlib) [ 2229.421366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2229.468282] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2238.177080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 2250.395929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.425319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2250.435679] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 2251.230182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2252.036149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.150@tcp [ 2252.443973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.468635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2252.492336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.516822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.538901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.572433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.601553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.630651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.667092] LNet: 121315:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2252.707119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2252.766465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2252.814435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2252.838548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2252.872927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2252.906536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.926480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2254.112774] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.150@tcp, 192.168.203.150@tcp1 [ 2254.287714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2254.334525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2254.390876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.450841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.505509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.556313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.631063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.667403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.767395] LNet: 121432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.798871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.857787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.906690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.949827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.982975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2255.019638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2255.773467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.150@tcp [ 2256.048169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2256.069966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2256.114541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2256.141776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2256.167384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2256.197349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2256.229963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2256.255960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2256.273216] LNet: 121549:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2256.294668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2256.324474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2256.360932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2256.406873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2256.444088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2256.481307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2257.055366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2257.306967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2257.356603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2257.406024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2257.450819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2257.498617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2257.531639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2257.561493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2257.588742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2257.627046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2257.653666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2257.700502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2257.739869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2257.773422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2257.800836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2262.418531] LNet: 121747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.425590] LNetError: 121747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.433659] LNet: Removed LNI 192.168.203.50@tcp [ 2262.441887] LNet: Removed LNI 192.168.203.50@tcp1 [ 2262.843200] Key type .llcrypt unregistered [ 2262.846116] Key type ._llcrypt unregistered [ 2279.738112] Key type ._llcrypt registered [ 2279.739596] Key type .llcrypt registered [ 2281.000642] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:33:53 (1787394833) [ 2288.846745] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:34:01 (1787394841) [ 2295.852169] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:34:08 (1787394848) [ 2303.252943] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:34:15 (1787394855) [ 2312.246158] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:34:24 (1787394864) [ 2320.141751] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:34:32 (1787394872) [ 2328.492475] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:34:41 (1787394881) [ 2336.474983] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:34:49 (1787394889) [ 2342.838119] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:34:55 (1787394895) [ 2349.514425] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:35:02 (1787394902) [ 2356.198266] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:35:09 (1787394909) [ 2365.259632] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:35:18 (1787394918) [ 2373.054354] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:35:25 (1787394925) [ 2380.324434] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:35:33 (1787394933) [ 2387.117873] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:35:39 (1787394939) [ 2393.615531] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:35:46 (1787394946) [ 2400.494961] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:35:53 (1787394953) [ 2407.560891] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:36:00 (1787394960) [ 2407.849751] LNet: 132286:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2407.849751] mask usage: [+|-] ... [ 2408.062283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.081975] alg: No test for adler32 (adler32-zlib) [ 2409.066401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.235686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2409.292675] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2410.342486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.365269] LNet: 132457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.376100] LNetError: 132457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.407951] LNet: Removed LNI 192.168.203.50@tcp [ 2410.422652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.513624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2410.552683] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2417.609658] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:36:10 (1787394970) [ 2424.570919] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:36:17 (1787394977) [ 2432.260397] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:36:25 (1787394985) [ 2439.233942] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:36:32 (1787394992) [ 2449.589602] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:36:41 (1787395001) [ 2458.191735] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:36:50 (1787395010) [ 2466.005100] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:36:58 (1787395018) [ 2473.934978] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:37:06 (1787395026) [ 2479.907904] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:37:13 (1787395033) [ 2486.236875] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:37:19 (1787395039) [ 2492.278536] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:37:25 (1787395045) [ 2498.808761] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:37:31 (1787395051) [ 2504.733888] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2269 sec ========== 06:37:37 (1787395057) [ 2505.131943] LNet: 139923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.139939] LNetError: 139923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.154160] LNet: Removed LNI 192.168.203.50@tcp [ 2505.754529] Key type .llcrypt unregistered [ 2505.760159] Key type ._llcrypt unregistered [ 2518.344224] Key type ._llcrypt registered [ 2518.356350] Key type .llcrypt registered [ 2518.528388] LNet: 140396:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2518.528388] mask usage: [+|-] ... [ 2518.754690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2518.770239] alg: No test for adler32 (adler32-zlib) [ 2519.788202] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2520.060943] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2521.768201] Key type lgssc registered [ 2522.960355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2596.840919] Lustre: Mounted lustre-client [ 2597.062947] LNet: 141871:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2597.062947] mask usage: [+|-] ... [ 2603.283313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2613.169761] Lustre: Unmounted lustre-client [ 2634.501657] Key type lgssc unregistered [ 2634.861360] LNet: 142854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.881486] LNetError: 142854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2634.916874] LNet: Removed LNI 192.168.203.50@tcp [ 2635.899160] Key type .llcrypt unregistered [ 2635.901904] Key type ._llcrypt unregistered